feat(dbt): extract dbt Jinja lineage and macros from raw .sql models

[alexisperinger-ux]

Summary

dbt .sql models are Jinja-templated, so without a compiled manifest a model and its dependencies are invisible to the graph: a referenced model like stg_users is not even a node. This indexes raw (uncompiled) dbt models:

• {{ ref('m') }} / {{ source('s','t') }} become USAGE lineage edges, via the vendored tree-sitter-jinja2 grammar.
• A dbt model (a .sql file with no macro defs) becomes a Model node keyed by file stem, so cross-file {{ ref('that_model') }} resolves into model-to-model lineage.
• {% macro name(...) %} becomes a Macro node.

Model is emitted only on the .sql path, so a plain .jinja / .j2 template is not treated as a model, and a macro-defining file is treated as a library, not a model.

No schema change: freeform Model / Macro labels and the existing USAGE edge type.

Index mode: macro extraction is gated to full mode, so the new nodes need a full-mode index.

Tests: dbt_jinja_macro_defs, dbt_jinja_ref_lineage, dbt_sql_ref_lineage in tests/test_extraction.c.

Complements the authoritative manifest path (#583).

Fixes #575.

Related PRs

This is one of three PRs that split the SQL + dbt graph-indexing work to keep each under the one-issue-per-PR contributing rule. They share the same extraction and registry plumbing, so they are one logical change reviewed as a set:

• #582: SQL DDL, first-class Table / View nodes + FROM/JOIN lineage (#574).
• This PR: dbt Jinja from raw .sql, Model / Macro nodes + ref() / source() lineage (#575).
• #583: dbt manifest ingestion, Model / Source nodes + DEPENDS_ON lineage (#576).

[DeusData]

Huge thanks for opening this PR and for the work you put into it.

The maintainer shop is currently full, so this may sit for a bit before it gets a proper review. We will come back to this as soon as possible with real feedback; I wanted to make sure it did not sit unacknowledged in the meantime.

[DeusData]

Reviewed as part of your set — the direction discussion lives on #582 and this covers what is specific to #584.

First, a structural thing you will want to know: this PR contains all of #582. Its first four commits are re-cherry-picks of the same four originals (same messages and cherry picked from trailers, different OIDs), and every hunk of #582's diff appears verbatim here. They are not a stacked branch — both target main directly, so they duplicate rather than build on each other. As it stands the two cannot both land: either this one goes in and #582 becomes redundant, or #582 lands and this gets rebased down to dbt-only. Worth deciding which you would prefer.

(Minor related signal: the tip commit is "resolve cherry-pick conflict markers in test_extraction.c". The final diff is clean — I checked — but conflict markers were pushed at some point.)

Security is clean, and I want to credit one choice specifically: you used the tree-sitter-jinja2 grammar already vendored on main rather than adding a dependency. That is the right instinct, and it means no MANIFEST.md obligation is triggered. The text scanner and AST walks are bounds-checked.

Three things give me pause, and they are about precision rather than safety.

1. The Model heuristic has no dbt gate. Any templated .sql file with zero macros becomes a Model node keyed by file stem. There is no dbt_project.yml detection, so an Airflow or Flyway repo — or any repo with Jinja-templated SQL — would grow false Model nodes. For a project whose first priority is graph accuracy, inventing nodes for non-dbt repos is the failure mode we care most about avoiding.

2. Macro recovery is a raw text scan. It is comment-blind, so a commented-out {% macro %} still emits a definition. In a tree-sitter codebase that is a precision outlier. I understand why — the grammar does not model {% %} blocks — but it is worth naming rather than leaving implicit.

3. The nested second parse establishes a pattern. Re-parsing a .sql file with a second parser inside cbm_extract_file is a new architectural shape for this codebase: a host language plus an embedded language. It works, and it is bounded. But patterns like this get copied, so if we want embedded-language support it deserves to be designed once deliberately rather than arriving ad hoc for one language pair. That is part of what I have put to the maintainer.

What is genuinely good here: raw dbt models really are code sitting in indexed repositories, they index today as broken SQL with zero lineage, and your design reuses the existing usage-resolution pass rather than inventing parallel machinery. The three dbt tests are binding. This is not a drive-by.

It also inherits #582's issues — the stale registry anchors and the third keep-in-sync site at pipeline_incremental.c:526 that neither PR touches, which on a naive rebase would mean labels registered during a full index but not on incremental reindex. With three labels now (Table, View, Model) the shared-registry short-name pollution surface is correspondingly larger.

Please hold off reworking until the maintainer answers the direction question on #582. This is the PR that most depends on that answer, and rebasing 462 lines against moved anchors is a poor use of your time before the concept is settled.