Rebase vendored Substrait proto from v0.23.0 to 0.98.0

[nielspardon]

Background

Gluten vendors its Substrait .proto files (gluten-substrait/src/main/resources/substrait/proto/substrait/) pinned near Substrait v0.23.0 with local patches, while upstream Substrait is now at 0.98.0. The JVM plan producer and both native consumers (Velox, ClickHouse) regenerate from this single proto source and ship together, and Gluten's Substrait plans are transient (built per query, serialized across JNI, discarded) — so there is no wire-compatibility constraint and the rebase is a pure code migration.

Goal

Bring the vendored proto to official 0.98.0 core (verbatim) plus Gluten's genuinely engine-specific additions grafted at non-colliding numbers, migrating the JVM producer and both native backends in lockstep.

Approach

Land as small, independently-green PRs, one per message family (each keeps protoc + JVM + both native builds compiling). Minimal rebase first — relocate only colliding fork fields and drop dead/decided-drop items; extracting engine-specific forks to formal Substrait extensions is deferred to a follow-up.

Increments

One PR per message family; each keeps protoc + JVM + both native builds green. Progress: 11 merged · Inc 8 to do (ReadRel split into 7a/7b/7c; 7a merged, 7b in review, 7c in review as two stacked PRs) · 1 resolved upstream.

• 0 — Remove unused vendored proto files + dead proto-based type derivation + Expression.Enum — merged ([GLUTEN-12597][CORE] Remove unused vendored Substrait proto files and dead proto-based type derivation #12598)
• 1a — plan.proto / extensions.proto → 0.98 (URI→URN) — merged ([GLUTEN-12597][CORE] Migrate Substrait extension referencing from URI to URN #12604)
• 1b — AdvancedExtension.optimization singular → repeated — merged ([GLUTEN-12597][CORE] Make AdvancedExtension.optimization repeated #12642)
• 2 — Type system: temporal → PrecisionTimestamp*; relocate Type.Nothing (kind 33→40, kept) — merged ([GLUTEN-12597][CORE] Migrate Substrait type system to 0.98 temporal types #12650)
• 3 — CrossRel → NestedLoopJoinRel (+ join-type enum remap) — merged ([GLUTEN-12597][CORE] Migrate nested loop joins from CrossRel to NestedLoopJoinRel (Substrait 0.98) #12700)
• 4 — Windowing: WindowRel → ConsistentPartitionWindowRel; WindowType → BoundsType — merged ([GLUTEN-12597][CORE] Migrate WindowRel to ConsistentPartitionWindowRel + BoundsType (Substrait 0.98) #12727)
• 5a — FetchRel offset/count → expressions — merged ([GLUTEN-12597][CORE] Migrate FetchRel offset/count to expressions (Substrait 0.98) #12704)
• 5b — AggregateRel.Grouping → expression references — merged ([GLUTEN-12597][CORE] Migrate AggregateRel.Grouping to expression references (Substrait 0.98) #12724)
• 5c — ExpandRel.advanced_extension — converged upstream via substrait-io/substrait#1167 (merged, ships 0.100.0); Gluten's graft already matches, no Gluten PR needed
• 6a — TopNRel → Expression count + offset + FetchMode — merged ([GLUTEN-12597][CORE] Migrate TopNRel to Expression count + FetchMode (Substrait 0.98) #12728)
• 6b — WriteRel: relocate bucket_spec (→ 1000); add create_mode / advanced_extension; OUTPUT_MODE_MODIFIED_TUPLES → _RECORDS — merged ([GLUTEN-12597][CORE] Migrate WriteRel to Substrait 0.98 (relocate bucket_spec; add create_mode/advanced_extension) #12746)
• 7 — ReadRel (the largest vendored message) — split into three independent slices (disjoint proto regions and consumers, can land in any order):
  • 7a — VirtualTable.values → Expression.Nested.Struct expressions — merged ([GLUTEN-12597][CORE] Migrate ReadRel.VirtualTable to Substrait 0.98 (values -> expressions) #12849)
  • 7b — text read-options redesign: TextReadOptions → DelimiterSeparatedTextReadOptions (map header/null_value to the 0.98 fields; graft max_block_size/empty_as_default at 1000+) — PR [GLUTEN-12597][CORE] Migrate ReadRel text read options to Substrait 0.98 (TextReadOptions -> DelimiterSeparatedTextReadOptions) #12917 (in review)
  • 7c — read_type → verbatim 0.98, in two stacked slices:
    • 7c-i — add official IcebergTable iceberg_table = 9; relocate the stream_kafka graft off the field-9 collision → 1000 — PR [GLUTEN-12597][CORE] Migrate ReadRel read_type to Substrait 0.98 (add iceberg_table, relocate stream_kafka) #12832 (in review)
    • 7c-ii — delete the stream_kafka graft entirely (making read_type verbatim 0.98) and remodel Gluten's ClickHouse Kafka read onto the official extension_table = 8 path: the StreamKafka payload moves to a new Gluten-owned kafka.proto, packed into ReadRel.ExtensionTable.detail and discriminated by the Any type_url (the pattern MergeTree/Range should converge onto — see [CH] Discriminate MergeTree/Range ExtensionTable reads by type_url, not isMergeTree=/isRange= marker strings #12842) — PR [GLUTEN-12597][CORE] Remodel Kafka read onto ReadRel.ExtensionTable (Substrait 0.98) #12841 (in review)
• 8 — Rel oneof reconciliation to exact 0.98 layout + drop RelRoot.output_schema — to do
• Docs (final) — after all increments land, reconcile docs/developers/SubstraitModifications.md with the finished proto: bump the stated base version v0.23.0 → 0.98.0 and drop entries for messages that ended up verbatim-upstream (e.g. ReadRel.VirtualTable), leaving only the genuinely remaining Gluten deviations

Upstream Substrait convergence PRs

Some kept forks converge by landing the equivalent field/type in Substrait core (substrait-io/substrait) rather than by dropping the fork. These are tracked under the apache-gluten label:

• substrait-io/substrait#1167 — advanced_extension on ExpandRel — merged, ships 0.100.0 → drives increment 5c; Gluten's graft already matches verbatim
• substrait-io/substrait#1147 — core Type.Null for untyped nulls — open → increment 2 relocates Type.Nothing to kind 40 to match; converge by renaming Nothing→Null once released (see note below)
• substrait-io/substrait#1138 — JSON read options on ReadRel.LocalFiles — open → upstreams Gluten's json read-options fork; increment 7b keeps the vendored json = 15 until this merges
• substrait-io/substrait#1139 — partition fields on ReadRel.LocalFiles — open → upstreams Gluten's partition-columns fork; the field-16 collision with the iceberg fork is a post-merge concern
• substrait-io/substrait#1140 — docs on typing untyped nulls — closed, superseded by substrait-io/substrait#1147

Tracked for later extraction waves (not driving a rebase increment):

• substrait-io/substrait#917 — GenerateRel for lateral view / unnest (@EpsilonPrime) — upstreams the pattern of Gluten's GenerateRel fork
• substrait-io/substrait#1105 — expression-valued window-aggregate bounds (@yongchul) — merged, ships 0.102.0; future convergence for Gluten's window frame bounds, mirroring the FetchRel offset→expression change

Note on Type.Nothing

Following the Substrait community sync (2026-07-29), the earlier plan to drop the Type.Nothing fork is superseded: upstream is adding a dedicated core null type (substrait-io/substrait#1147 — Type.Null at kind field 40). Increment 2 relocates Nothing to field 40 to match, so once that PR releases the fork converges by renaming the message Nothing→Null — no field renumbering, no consumer-mapping changes. Gluten keeps Type.Nothing until then.

Related: upstream consolidation work is tracked under the apache-gluten label in substrait-io/substrait.

[nielspardon]

Update — Type.Nothing direction changed (Increment 2)

The Increment 2 checklist item previously read "drop Type.Nothing". That's been corrected to relocate (kind 33→40, kept).

Rationale: in the Substrait community sync on 2026-07-29 the PMCs decided upstream should add a dedicated core null type rather than have producers avoid a bottom type. That proposal is up as substrait-io/substrait#1147 (Type.Null at kind field 40). Increment 2 deliberately relocates Gluten's Nothing fork to field 40 to match, so once #1147 is merged and released upstream the fork converges by a pure message rename Nothing→Null — no field renumbering and no changes to the Velox/ClickHouse consumer mappings. Gluten keeps Type.Nothing as-is until then.

Net effect: Increment 2 is semantically neutral for Nothing (relocate only), and the former "drop" step becomes a future rename gated on the upstream Type.Null release.