Skip to content

feat(attachments): multi-annex embed for Sales Invoice (part 1) - #263

Open
dafrose wants to merge 37 commits into
alyf-de:version-16-hotfixfrom
dafrose:feat/embed_attachments_16
Open

feat(attachments): multi-annex embed for Sales Invoice (part 1)#263
dafrose wants to merge 37 commits into
alyf-de:version-16-hotfixfrom
dafrose:feat/embed_attachments_16

Conversation

@dafrose

@dafrose dafrose commented Jun 15, 2026

Copy link
Copy Markdown
Member

Summary

Part 1 of selectable PDF annexes for Sales Invoice (#261, coordination #76). This PR is supposed to only add support for multiple attachments via the same embedding path as before. File attachments are still embedded into XML (which may be embedded into PDF). It does not provide support for embedding additional files directly into PDF. This is planned for a follow-up PR.

When Multiple Attachment Embedding is enabled on E Invoice Settings, annexes are selected from the new einvoice_attachments child table and embedded in the e-invoice XML as 916 supporting documents (same _embed_attachments loop as today). When the setting is off, behaviour stays on the legacy einvoice_embedded_document attach field.

  • einvoice_attachments child table on Sales Invoice with Desk attach button and first-row-wins dedup on validate
  • multi_attachment_embed_enabled checkbox on E Invoice Settings
  • get_embed_attachments branches between table and legacy field; shared module sales_invoice_attachments.py
  • Migration on enable: per-save move of legacy field → table (orange msgprint); bulk RQ job via Desk button; legacy field hidden + read_only while multi-embed is on
  • Embed refactor: get_legacy_embed_attachment + _embed_attachments URL loop (no change to hybrid PDF mechanics yet — annex bytes still travel inside embedded Factur-X XML)
  • Tests: New tests cover embedding behaviour of the legacy path as well as the table path, migration from legacy field to table and table-specific validations.

Not in this PR (possible follow-ups): PDF/A-3 factur-x pipeline with N PDF file attachments (#262), profile-aware 916 encoding, MIME allowlist, v15/develop backports.

Settings:
image

Migration Button UI:
image

Sales Invoice EInvoice tab:
image

Test plan

  • bench --site <site> run-tests --app eu_einvoice --lightmode — 22 tests green
  • E Invoice Settings: enable Multiple Attachment Embedding → legacy attach field hidden/read-only
  • Sales Invoice with legacy einvoice_embedded_document only → save with setting on → row appears in Embedded Documents, legacy field cleared, orange migration message
  • Sales Invoice with broken legacy URL → save with setting on → field unchanged, Error Log entry (no blocking save)
  • E Invoice SettingsMigrate attachments to table → RQ job clears legacy fields site-wide
  • Sales Invoice with two rows in einvoice_attachments → submit/download PDF → annex payloads present in embedded Factur-X XML (Desk spot-check or XML download)

Notes for reviewers

  • Hybrid PDF path unchanged: attach_xml_to_pdf still embeds invoice XML only; new integration test verifies table annex content survives the PDF round-trip inside that XML.
  • Target branch is version-16-hotfix per rollout plan; develop backport differs (patch migration drops legacy field).

Ticket Ref

LMK-16

@dafrose
dafrose force-pushed the feat/embed_attachments_16 branch from 3e495c6 to 53520f4 Compare June 15, 2026 12:17
@dafrose

dafrose commented Jun 16, 2026

Copy link
Copy Markdown
Member Author

@greptileai

@greptile-apps

greptile-apps Bot commented Jun 16, 2026

Copy link
Copy Markdown

Confidence Score: 5/5

  • Safe to merge; all changes are well-contained and the previously reported issues have been resolved.
  • The new embedding path is thoroughly tested, permission checks are in place, the bulk migration uses per-invoice commits with rollback on failure, and the validate_doc gating correctly limits migration and row validation to the multi-embed mode. The two flagged items are edge-case nits that do not affect the main code paths.
  • eu_einvoice/european_e_invoice/custom/sales_invoice_attachments.py — the _file_already_in_attachments parameter naming and the duplicate-filename migration edge case are both minor but worth a quick look before the next iteration.

Sequence Diagram

sequenceDiagram
    participant Desk as Sales Invoice (Desk)
    participant VD as validate_doc
    participant MIG as migrate_legacy_embed_to_table
    participant VAL as validate_einvoice_attachment_rows
    participant GEN as EInvoiceGenerator._embed_attachments
    participant GEA as get_embed_attachments
    participant DB as Database

    Desk->>VD: save / submit

    VD->>DB: get_single("E Invoice Settings")
    DB-->>VD: settings

    alt "multi_attachment_embed_enabled = True"
        VD->>MIG: migrate_legacy_embed_to_table(doc)
        MIG->>DB: _resolve_embed_file_for_invoice
        DB-->>MIG: "File | None"

        alt File resolved
            MIG->>MIG: _persist_legacy_embed_migration_on_save
            MIG-->>VD: True (migrated)
        else Broken URL, table has rows
            MIG->>DB: clear einvoice_embedded_document (in-memory)
            MIG-->>VD: False
        else Broken URL, no table rows
            MIG->>DB: log_error
            MIG-->>VD: False
        end

        VD->>VAL: validate_einvoice_attachment_rows(doc, settings)
        VAL->>VAL: _validate_duplicate_embed_filenames
        VAL->>DB: "get_value(File, file_id, attached_to_*)"
        VAL->>DB: get_value(File, file_id, content_hash) per row
        VAL-->>VD: ok / throw ValidationError
    else "multi_attachment_embed_enabled = False"
        VD->>VD: skip migration and table validation
    end

    VD->>VD: validate_einvoice(doc) if configured

    Note over GEN,GEA: On e-invoice generation (download / submit)

    GEN->>GEA: get_embed_attachments(invoice)
    GEA->>DB: get_single_value(multi_attachment_embed_enabled)

    alt "enabled = True"
        GEA->>GEA: _get_table_embed_attachments (validates again)
        GEA-->>GEN: [EmbedAttachment, ...]
    else "enabled = False"
        GEA->>GEA: _get_legacy_embed_attachment
        GEA-->>GEN: "[EmbedAttachment] | []"
    end

    loop for each EmbedAttachment
        GEN->>DB: frappe.get_doc("File", attachment.file)
        DB-->>GEN: File doc
        GEN->>GEN: build AdditionalReferencedDocument (ARD 916)
    end
Loading

Reviews (22): Last reviewed commit: "refactor(install): remove redundant afte..." | Re-trigger Greptile

Comment thread eu_einvoice/european_e_invoice/doctype/e_invoice_settings/e_invoice_settings.py Outdated
Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice_attachments.py Outdated
Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice_attachments.py Outdated
@dafrose

dafrose commented Jun 17, 2026

Copy link
Copy Markdown
Member Author

@greptileai

@dafrose
dafrose marked this pull request as ready for review June 17, 2026 08:10
@dafrose
dafrose requested a review from barredterra June 17, 2026 08:11
Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice_attachments.py Outdated
@dafrose
dafrose force-pushed the feat/embed_attachments_16 branch from d3710c5 to d0582ca Compare June 17, 2026 09:08
Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice_attachments.py Outdated
@dafrose
dafrose force-pushed the feat/embed_attachments_16 branch from d0582ca to a5c5022 Compare June 17, 2026 13:27
@barredterra
barredterra requested a review from 0xD0M1M0 June 19, 2026 00:53
Comment thread eu_einvoice/european_e_invoice/doctype/e_invoice_settings/e_invoice_settings.py Outdated
Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice.js
@dafrose
dafrose force-pushed the feat/embed_attachments_16 branch from 077bfb3 to 49cfcc5 Compare June 19, 2026 09:56
Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice_attachments.py Outdated
@greptile-apps

greptile-apps Bot commented Jun 19, 2026

Copy link
Copy Markdown

Want your agent to iterate on Greptile's feedback? Try greploops.

@dafrose

dafrose commented Jun 19, 2026

Copy link
Copy Markdown
Member Author

I tested a simple print format addition for the embedded files.

The code I used:

		{# Embedded e-invoice annexes (916 supporting documents) #}
		{% set multi_embed = frappe.db.get_single_value("E Invoice Settings", "multi_attachment_embed_enabled") %}
		{% if multi_embed and doc.einvoice_attachments %}
		<div class="info-card">
			<div class="title">{{ _("Embedded Documents") }}</div>
			<ul class="small-text" style="margin: 8px 0 0 0; padding-left: 18px;">
				{% for row in doc.einvoice_attachments %}
				{% set file_url = frappe.db.get_value("File", row.file, "file_url") %}
				<li style="margin-bottom: 4px;">
					{% if file_url %}
					<a href="{{ frappe.utils.get_url(file_url) }}">
						{{ row.display_name or row.file_name or file_url }}
					</a>
					{% else %}
					{{ row.display_name or row.file_name or row.file }}
					{% endif %}
				</li>
				{% endfor %}
			</ul>
		</div>
		{% elif not multi_embed and doc.einvoice_embedded_document %}
		<div class="info-card">
			<div class="title">{{ _("Embedded Document") }}</div>
			<p class="small-text" style="margin-top: 8px;">
				<a href="{{ frappe.utils.get_url(doc.einvoice_embedded_document) }}">
					{{ doc.einvoice_embedded_document }}
				</a>
			</p>
		</div>
		{% endif %}

The result:

image

Comment thread .gitignore Outdated
@dafrose
dafrose force-pushed the feat/embed_attachments_16 branch 3 times, most recently from 087b3fb to 73b240e Compare June 26, 2026 12:03
Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice_attachments.py
Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice.py Outdated
@dafrose

dafrose commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

@greptile-apps

@dafrose

dafrose commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

Update since 19 Jun 2026

Focus: annex validation hardening, Desk field exclusivity, and two production fixes.

What changed

  • Unique embed filenames (BR-DE-22) — DB unique index on (parent, file_name) on E Invoice Attachment Row; save-time duplicate check when an error action is configured; embed/XML path always enforces uniqueness.
  • Duplicate annex content — identical File content_hash across table rows → orange Desk hint on save (warn-only; filenames can still differ).
  • Desk exclusivity — with multi-embed off: only legacy Embedded Document visible; with multi-embed on: only Embedded Documents table; legacy hidden + read-only. Synced from E Invoice Settings, after_install, and after_migrate.
  • Annex row validationvalidate_einvoice_attachment_rows runs only when multi_attachment_embed_enabled is on (orphan table rows in legacy mode no longer block save).
  • Bulk legacy migrate — failed per-invoice DB persist rolls back the transaction so a later success cannot leave orphan child rows.
  • Binary annex embed — CII embed reads file bytes with get_content(encodings=[]) so small PNG/PDF annexes are not corrupted by Frappe’s default text decoding.
  • German localede.po / main.pot updated for new strings; run compile-po-to-mo if you test Desk in German.

Tests: bench --site <site> run-tests --app eu_einvoice --lightmode (app tests self-seed; no ERPNext test_records preload).

Reviewer setup — unique constraint

There is no migrate patch for unique_parent_file_name. Greenfield installs get the constraint when the child DocType is first synced.

If your review site already had E Invoice Attachment Row before pulling this branch:

  1. Remove any duplicate (parent, file_name) rows on test invoices.
  2. Apply the constraint once:
bench --site <site> execute eu_einvoice.european_e_invoice.doctype.e_invoice_attachment_row.e_invoice_attachment_row.on_doctype_update

bench migrate alone does not re-run on_doctype_update when the child DocType JSON hash is unchanged.

Alternatively: fresh site + install-app eu_einvoice, or save E Invoice Attachment Row in Desk once to trigger on_doctype_update.

Test plan

Area Steps
Constraint Enable multi-embed; on one draft Sales Invoice, add two Embedded Documents rows with the same File Name → save blocked with duplicate-filename message when error action is set.
Content hash Two rows, different filenames, same file content → orange msgprint; save still allowed.
Exclusivity Never-enabled site: reload invoice form → only legacy attach. Enable setting → reload → only table; legacy not editable.
Validate gate (Optional) With setting off and table rows present via API/import → save does not run annex row validation (normal Desk path hides the table).
Bulk migrate E Invoice Settings → migrate with Include submitted; submitted invoice with legacy embed migrates to table; broken URL skip/remove still works.
Embed smoke Draft invoice with one annex (legacy or table path) → create/download e-invoice XML; annex present as ARD 916.

Automated: bench --site <site> run-tests --app eu_einvoice --lightmode.

@barredterra barredterra left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested merge bar

Solid after the earlier review rounds (permissions, POST-only, orphaned-row rollback, binary encodings=[], field exclusivity). CI is green. Please address these three correctness items before merge:

  1. Don’t use validating save() for bulk draft migration (or skip e-invoice validation there).
  2. Always enforce duplicate annex filenames when multi-embed is on.
  3. Set idx on direct child inserts.

Inline comments below.

Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice_attachments.py Outdated
Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice_attachments.py Outdated
Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice_attachments.py
Comment thread eu_einvoice/custom_fields.py
Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice_attachments.py Outdated
Comment thread eu_einvoice/european_e_invoice/custom/sales_invoice_attachments.py
dafrose and others added 28 commits September 7, 2026 08:19
Add an Include submitted Sales Invoices option on the E Invoice Settings
migrate button. Draft invoices still migrate via save; submitted invoices
use direct child-row inserts and db.set_value so post-submit restrictions
do not block the job. Replace the bulk migration integration test with a
draft/submitted × include_submitted matrix and cancel submitted fixtures
before teardown.
Throw when an einvoice_attachments row points at a File without a
file_url, load File rows via the normal get_doc path during legacy
embed resolution, and restrict site-wide migration to System Manager.
Restructure attachment test coverage: split get_embed_attachments
branching into subTest cases, drop redundant table-principles unit class,
extract URL-order resolver checks, and restore multi-embed settings in
integration setUp/tearDown. Move set_multi_attachment_embed_enabled and
assert_single_orange_message into tests/helpers.py.

Add integration coverage for bulk migration of cancelled submitted
invoices and prefer a positive docstatus check in bulk_migrate_legacy_embed_attachments.

Regenerate fixtures to update attachment row schema and enable file id field visibility,
for the row filter to work on Sales Invoice Attachments table.
Share hidden/read_only flags via legacy_embed_field_lockdown_properties()
so get_custom_fields and set_legacy_embed_field_lockdown stay aligned. This
prevents after_install and migrate patches from re-exposing the legacy attach
field when multi-attachment embedding is enabled.
Give operators explicit feedback when bulk legacy-embed migration hits
unresolvable URLs: skip (Error Log, field unchanged) or remove (clear field,
site warning log). Add a dialog toggle, split the job summary into skipped,
removed, and error counts, and restrict the enqueue API to POST.
Show an orange msgprint when per-save migration cannot resolve
einvoice_embedded_document, in addition to the existing Error Log.
Guide invoice owners to ask a System Manager to run bulk migration on
E Invoice Settings. Remove unused _log_removed_broken_legacy_embed
(inlined in _handle_broken_legacy_embed). Add tests for the validate
warning and bulk migration summary segments.
Expand Google-style docstrings on embed resolution, legacy migration,
bulk migrate APIs, and test helpers. Regenerate locale catalogs and
translate milestone 1 user-facing strings in de.po for the multi-annex
embed settings, migration dialog, and Sales Invoice annex grid.
…ution

Run setup_complete with company and fiscal-year args so lightmode CI
seeds ERPNext without wizard insert errors. Prefer the legacy attach-field
File row when submit sync created duplicate URLs for the same path, and
align bulk migration tests with Desk attach behaviour.
…le id

Replace deduplicate_attachment_rows with save-time
validate_einvoice_attachment_rows: duplicate file_name respects E Invoice
Settings error action; identical content_hash is warn-only.

Resolve embeds via EmbedAttachment (File id + file_name). validate_attachments
blocks duplicate filenames and missing File rows at generation time.
_embed_attachments uses row.file_name for ARD 916 @filename (BR-DE-22).
Add a composite unique constraint on E Invoice Attachment Row for parent and file_name
so duplicate embed filenames cannot be persisted for one Sales Invoice.
Cover the schema hook with a database-specific constraint test.
…d XML

Share case-insensitive duplicate detection between validate_einvoice_attachment_rows
and _get_table_embed_attachments; remove validate_attachments and the warn-only
path so save-time checks align with the DB constraint. Require file_name on
E Invoice Attachment Row and document display_name as print-only.
…bility

Show only the legacy attach field before multi-embed is enabled, and only
the Embedded Documents table afterward. Sync Custom Field flags from
settings updates, after_install, and after_migrate, and update German
locale strings.
…ex bytes

Rollback the transaction when site-wide legacy embed migration fails mid-persist
so a later commit cannot flush orphaned child rows. Read embedded annex files with
get_content(encodings=[]) so PNG/PDF bytes are not text-decoded before base64.
Add regression tests and stabilize embed test fixtures when pdf_on_submit is
installed.
Route all successful bulk legacy-embed migrations through
_persist_legacy_embed_migration_db so unrelated Sales Invoice
validation cannot block drafts when Action on Validation Error
during Save is enabled.
BR-DE-22 uniqueness is a format rule, not an e-invoice error-action
toggle. Run duplicate-filename validation whenever multi-embed annex
rows are validated, including on Postgres where the unique index is
case-sensitive.
Assign max(idx) + 1 when bulk legacy migration inserts E Invoice
Attachment Row records so Desk grid order is stable instead of
defaulting to idx 0.
Co-authored-by: Raffael Meyer <14891507+barredterra@users.noreply.github.com>
…mbeds

Queue site-wide bulk migration when Multiple Attachment Embedding is first
enabled on E Invoice Settings. Fail XML/download when the legacy attach field
is still set so annexes cannot silently drop after the setting flip.

Bulk migration always covers all docstatus via direct DB persist; remove the
include_submitted option from the API and migrate dialog. Share enqueue logic
through queue_bulk_migrate_legacy_embed_attachments and publish progress and
completion updates to the requesting user via realtime msgprint.
Validate table embed rows against File.attached_to_* on save and at XML
generation so API/import paths cannot reference arbitrary File ids. Add unit
and integration coverage; attach test annex files to invoices in helpers.
Drop the site-wide file_url fallback in _resolve_embed_file_for_invoice.
A shared URL on another Sales Invoice could be migrated into the child
table and then fail validate_einvoice_attachment_rows, blocking every
save until manual DB intervention.

Only consider File rows attached to the current invoice; treat unmatched
legacy URLs as broken links. Add an integration test for the cross-invoice
URL collision case.
Only block embed when the legacy field is set and the annex table is
empty. On save, clear unresolvable legacy URLs when table rows already
exist so embed can proceed from the table.
Skip inserting an E Invoice Attachment Row when the resolved File is
already listed in einvoice_attachments; still clear einvoice_embedded_document
on bulk migrate and on-save. Count bulk skips as already_migrated and
suppress the orange “moved” msgprint when no row was appended.

Add integration coverage for bulk and validate paths. Regenerate POT/PO
and translate the broken-legacy-link-cleared message plus workspace labels.
Embedded Documents cannot be filled until the invoice is saved (File needs
attached_to_name). Hide Add Row via cannot_add_rows while is_new so users
are not offered empty, unusable rows.
@dafrose
dafrose force-pushed the feat/embed_attachments_16 branch from 834a1d6 to eaf3eee Compare September 7, 2026 06:25
Custom field exclusivity is applied once via the make_custom_fields patch
and on E Invoice Settings save; drop the per-migrate hook barredterra
flagged as redundant on PR alyf-de#263.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants