Extract only what changed and propagate upstream deletions

[omakszewska]

Why

Extraction ran over the whole staging directory on every run. Idempotency existed but started too late — _compute_page_hash works on extracted text, so the LLM and graph stages were skipped while the OCR that produced their input had already been paid for. And nothing propagated deletions: a document withdrawn upstream stayed in the graph, so the assistant kept answering from sources that no longer exist. Both need the trigger to carry more than changed_count > 0, so they ship together.

What this does

refresh_sources_flow passes changed and deleted path sets to data_pipeline_flow, which filters the staging scan before ocr_extraction. changed=None still means a full scan — deliberately, since that is the only way to pick up extractor or OCR_ changes. It is also why no file-level cache was added: hashing input bytes never invalidates when the extractor changes, hashing output does.

Nodes now record where they came from. Variables are recovered from the generated MERGE clauses and a FROM_SOURCE tail is appended to the same statement, with the source id as a query parameter — a relationship rather than a list property, because concurrent workers appending to a list lose updates, and a lost entry later means deleting a node that still has a source. The deletion stage detaches a document's Source nodes (prefix match, since one file maps to many page ids), removes entities left with no source, archives the staged file to {staging}/.archive, and drops the manifest entry only for ids the pipeline confirmed.

Failure handling is what makes the rest safe. data_pipeline_flow returns what it confirmed instead of relying on the absence of an exception: OCR swallows per-file errors and the flow only warns on failed pages, so a broken PDF would otherwise be marked processed and never retried once extraction is filtered. Manifest entries carry pending/processed/failed and are retried up to DATA_PIPELINE_MANIFEST_MAX_ATTEMPTS; a byte change re-arms a failed entry. Deletions come only from the explicit set, never from diffing a run's output — a document that failed to extract produces zero pages and is indistinguishable from one that is gone — and are refused when discovery returned nothing or names an implausible share of the manifest.

Nodes ingested before provenance existed have no FROM_SOURCE edge and are not cleanable; rebuilding once via graph_dump.py would fix it.

Tests: 287 → 312, 745 of the 1331 added lines.

Closes #49

[Dominik-Galus]

uv run dedup-graph corrupts provenance, INTERNAL_LABELS = {"ProcessedDocument", "PipelineRun"}. The new Source label isn't in it, and isn't in the 27-label vocabulary. So the full dedup walk's relabel_off_vocabulary_nodes treats every Source node as off-vocabulary drift. So uv run dedup-graph runs MATCH (n:Source) REMOVE n:Source SET n:Topic on every provenance node. After that: delete_sources_for_documents's MATCH (s:Source) finds nothing, and the ex-Source nodes become Topic entities polluting retrieval.

[Dominik-Galus]

In rag FULLTEXT_EXCLUDED_LABELS also omits Source. Harmless in practice because Source has no title/context, so it never matches, but it forces an index rebuild on the label-set change and pollutes the index label list.

[Dominik-Galus]

two documents with byte-identical extracted page text still share entities whose FROM_SOURCE points only at the first document, deleting the first orphans entities the second still cites but this is a really unlikely edge case, so we can merge it or if you want you can extend it but it is not necessary