clarion-pdb

Generate Microsoft PDB files for Clarion (TopSpeed) binaries so they can be debugged in modern Windows debuggers — WinDbg (+ Time-Travel), Visual Studio, VS Code — and used for crash-dump / ETW symbolication. Clarion ships its own proprietary TSWD debug info (no CodeView/PDB); this project translates it.

Full reverse-engineering analysis: docs/TestDashboard-DebugInfo-Analysis.md Plan, decisions, progress: PDB-Generation-Plan.md

Why this matters

Clarion's debug information is locked to its own debugger. The PDB is the format the entire modern Windows toolchain speaks — once a Clarion binary has a valid PDB, that toolchain opens up to it: stepping Clarion source in VS Code and Visual Studio, walking call stacks (and Time-Travel Debugging) in WinDbg, and symbolicating crash dumps / ETW traces with the same names and line numbers you wrote.

This repo proves that translation is achievable and already produces PDBs that resolve in those tools. It's the foundational piece: turning "achievable" into a polished, push-button experience for any Clarion developer will take more work — packaging the emitter into a single zero-dependency exe, fuller type coverage (classes/methods/VMT), and editor-debugger glue (a VS Code launch config, source mapping). The hard part — producing a PDB the standard tools actually accept — is done; the rest is tooling around it.

Status — working end to end (tests 12/12)

From a linked Clarion EXE/DLL (32-bit PE) plus its linker .MAP, clarion-pdb produces a real PDB that resolves in WinDbg / VS / llvm-symbolizer:

In the PDB	Source	Verified
Publics	linker MAP	0x4026B4 → VIEWHISTORY@F10ERRORCLASS
Procedures (S_GPROC32)	MAP + section contributions	1,472
Source lines	TSWD +0x1C address→line table (module-tagged, 100% .text)	0x401480 → ABERROR.CLW:1221
Types (LF_STRUCTURE)	OMF 0xD3 per-module type grammar	GROUP/QUEUE members byte-exact
Typed globals (S_GDATA32)	TSWD +0x2C typed-symbol tree	COLORMAP → COLOR@0, TEXT@4, TEXTKEY@29

The real PDB writer is LLVM's PDBFileBuilder (C++), which emits the DBI section contributions + TPI hashes that the earlier yaml2pdb spike could not — so lines and procedures actually resolve, not just appear.

Use

# one command: EXE in -> PDB out + an RSDS-bound copy of the EXE to debug
python src/clarion_pdb.py path\to\App.exe
#   --map M       linker .MAP (default: auto-located next to the EXE)
#   --obj O       a module .obj to mine GROUP/QUEUE types from (optional)
#   --pdb OUT     output PDB (default: <stem>.pdb beside the EXE)
#   --out-exe E   RSDS-bound EXE copy (default: <stem>.dbg.exe); --no-patch to skip
#   --guid G      RSDS GUID (default: freshly generated; threaded to PDB + patch)

--guid/--age are threaded through both the PDB and the PE patch, so the binary always binds to the PDB built with it. Debug the *.dbg.exe.

Dependencies to use the tool: Python 3 and the prebuilt clarion-pdb-emit.exe (LLVM is statically linked into it — no separate LLVM install). You debug in the tools you already have (WinDbg / VS). The LLVM command-line tools (llvm-pdbutil, llvm-symbolizer) are not used by the tool — only by the test suite to verify output. (Goal: fold the Python front-end + the native emitter into one self-contained exe so there are zero deps — see "Remaining".)

python tests/test_pipeline.py          # regression suite — needs LLVM tools (scoop)
python src/cdm/ir.py <exe> <map> out.ir.json <obj>   # just the CodeView IR

Debug in VS Code

VS Code reads PDBs through the C/C++ extension (ms-vscode.cpptools) using its cppvsdbg debugger (the Visual Studio debug engine — Windows only). Generate the PDB first (above), then point a launch config at the RSDS-bound *.dbg.exe — it carries the GUID that binds it to the matching PDB:

// .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug Clarion app (PDB)",
      "type": "cppvsdbg",     // requires ms-vscode.cpptools; Windows only
      "request": "launch",
      "program": "${workspaceFolder}/App.dbg.exe",   // the RSDS-bound copy
      "cwd": "${workspaceFolder}",
      "stopAtEntry": false,
      "symbolSearchPath": "${workspaceFolder}"        // where App.pdb lives
    }
  ]
}

Keep App.pdb next to the .dbg.exe (or on symbolSearchPath). Set breakpoints in your .clw/.inc source and step by Clarion line.

Caveat — this is the editor glue still being smoothed out. Source-line stepping depends on the PDB's file paths matching where your sources sit; if they differ you'll need a source remap (e.g. sourceFileMap). Watch/locals display improves with the type and NatVis work tracked under Remaining. For raw stack/line resolution today, WinDbg and llvm-symbolizer are the most reliable.

Build the emitter (one-time)

The emitter links LLVM dev libraries (the scoop LLVM is tools-only); built via the vcpkg bundled with Visual Studio:

& "...\VC\vcpkg\vcpkg.exe" install llvm:x64-windows         # heavy, one-time
cmake -S src/backend -B build/backend -DCMAKE_TOOLCHAIN_FILE=F:/vcpkg/scripts/buildsystems/vcpkg.cmake
cmake --build build/backend --config Release                 # -> clarion-pdb-emit.exe

Layout

src/
  clarion_pdb.py         the single-command driver (build_ir -> emit -> rsds_patch)
  omf/omf_reader.py      OMF object reader (records + 0xD3 debug COMENTs)
  omf/d3_decoder.py      OMF 0xD3 type/member grammar -> structs (GROUP/QUEUE)
  pe/pe_sections.py      PE32 section table + RVA->(seg,off)
  pe/rsds_patch.py       add an RSDS/CodeView debug entry to a PE copy
  map/map_reader.py      linker MAP: by-address publics + segments
  tswd/tswd_lines.py     TSWD +0x1C address->line table (module-tagged)
  tswd/tswd_symbols.py   TSWD +0x2C typed-symbol tree -> static data globals
  cdm/ir.py              CodeView IR (JSON) — the front-end/back-end contract
  backend/pdb_emit.cpp   LLVM PDBFileBuilder emitter (ir.json -> .pdb)
tests/test_pipeline.py   regression suite (12 tests)
docs/                    analysis, plan, research

Remaining

• Single self-contained exe (statically bundle the emitter — zero deps for the user).
• v4 classes / methods / VMT (LF_CLASS, vtables, base classes).
• v5 NatVis (DECIMAL / CSTRING / QUEUE display) + signature demangling.
• Type the procedures (S_GPROC32.FunctionType); decode all .objs; recurse nested/DIM'd GROUP fields.

License

MIT © Mark Sarson