[lake/paimon] Support clean and legacy Paimon lake table schemas

[fhan688]

Purpose

Linked issue: close #3902

Sub-task of the FIP-27 umbrella (#2411): Remove Mandatory System Columns From Fluss Lake Tables.

Today every Paimon lake table Fluss creates is forced to carry three mandatory system columns (__bucket, __offset, __timestamp) as its last physical columns. They pollute the schema users see from Paimon and other engines, and impose a system-metadata-based physical layout.

This PR implements the Paimon part of FIP-27: newly created Paimon lake tables use a clean physical schema containing only user-defined columns, while existing legacy tables that still carry the three system columns remain fully readable and writable without any schema migration. Both layouts are supported across create, tiering writers, readers, projections, schema evolution, and re-enabling tiering.

Brief change log

• Layout detection (single source of truth): add PaimonSystemColumns with a LakeLayout enum and detectLayout(RowType). Detection is purely schema-based — no new table property or metadata, so existing tables are never migrated:
  • no system columns → CLEAN;
  • all three present, in canonical order, as the trailing columns with compatible types → LEGACY;
  • partial / out-of-order / type-incompatible system columns → rejected with a clear InvalidTableException.
• Create: PaimonConversions.toPaimonSchema no longer appends the system columns; new tables are clean. The user-column name-conflict check against system names is kept.
• Schema evolution: toPaimonSchemaChanges now takes the target layout — for a legacy table a new business column is still inserted before the first system column; for a clean table it is appended normally.
• Compatibility / re-enable tiering: PaimonTableValidation.isPaimonSchemaCompatible detects the existing layout and, for a legacy table, enriches the freshly generated clean schema with the trailing system columns before comparison, so disabling and re-enabling tiering preserves the existing physical layout. The __timestamp precision-6→3 relaxation is guarded so it only applies to legacy tables (a clean table has no __timestamp).
• Row tiering writer: FlussRecordAsPaimonRow emits the three system values only for legacy tables; for clean tables the business-field count equals the full row and no system fields are written. Layout is threaded through PaimonLakeWriter → RecordWriter → AppendOnlyWriter /MergeTreeWriter.
• Arrow-batch tiering writer: for clean tables AppendOnlyArrowBatchHelper writes the original VectorSchemaRoot directly; for legacy tables it keeps enriching the batch with the __bucket/__offset/__timestamp vectors.
• Reader / projection: PaimonRecordReader projects and reads __offset/__timestamp only for legacy tables. For clean tables it emits a sentinel -1 log offset / timestamp, consistent with the existing LakeRecordRecordEmitter contract (logOffset() >= 0 marks the incremental phase) and the existing UNKNOWN_OFFSET = -1 convention.
• Row adapter: PaimonRowAsFlussRow no longer assumes a fixed number of trailing system columns globally; the trailing-system-column count is explicit, which also fixes latent miscounts for nested/projected rows (e.g. the lookup path, where Paimon already projects system columns away).

Tests

• Adapted FlussRecordAsPaimonRowTest to the layout-aware writer (existing cases are legacy-layout).
• TODO before merge — new clean-layout coverage: a newly created table exposes only user columns; legacy tables keep tiering and are readable with Flink FULL startup; disable+re-enable preserves layout; row and Arrow-batch writer paths; reader projections; schema evolution; and rejection of a partial/type-incompatible legacy layout. Cases should span log and primary-key tables, and partitioned and non-partitioned tables.
• Build note: mvn clean verify has not been run locally (JDK 11 build environment not available on my machine; only JDK 8). Verified signature consistency, imports, and static review; full compile + IT run is pending on a JDK 11 environment.

API and Format

No public API change. This changes the physical schema of newly created Paimon lake tables (clean layout by default). Existing tables are not migrated and keep their current physical format. Compatibility / rolling-upgrade requirements are covered by the umbrella #2411 and documented in #3905:

• New lake storage plugins and Flink connectors continue reading legacy tables.
• New tiering services keep writing the legacy layout when the target table has the system columns.
• Old tiering services must not process newly created clean tables; old Flink connectors using
  FULL startup mode must not read newly created clean tables.
• Safe upgrade order: lake-reading Flink connectors and lake storage plugins → tiering service →
  Fluss cluster.

Documentation

Feature behavior (clean vs. legacy layouts, detection, and the rolling-upgrade/compatibility matrix) is documented separately under #3905. No standalone doc change in this PR.