TCP I5: Add Nested(...) support - #455
Draft
alex-clickhouse wants to merge 8 commits into
Draft
Conversation
TriageCategory: Summary What this impacts
Concerns
Required reviewer action
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds TCP/Native client support for ClickHouse Nested(...) when it is carried as a single wire column (flatten_nested = 0), by introducing a dedicated columnar representation and codec (rather than reusing Array(Tuple(...))).
Changes:
- Register a new
Nestedtype codec in the TCPColumnCodecRegistry. - Implement
NestedColumnCodecto read/write theNestedwire format (offsets + one flattened stream per field), producing/consuming a denseNestedColumn. - Add unit + integration-style insert round-trip coverage for
Nested(...), including >7 fields and composite field types.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| ClickHouse.Driver.Tcp/Types/NestedColumn.cs | New dense column type that stores per-field flat columns plus shared per-row offsets, with columnar access and cached row materialization. |
| ClickHouse.Driver.Tcp/Types/ColumnCodecRegistry.cs | Registers Nested codec factory in the default TCP codec registry. |
| ClickHouse.Driver.Tcp/Types/Codecs/NestedColumnCodec.cs | New codec implementing Nested(...) single-column wire layout (byte-identical to Array(Tuple(...))), without tuple arity limits. |
| ClickHouse.Driver.Tcp.Tests/Utilities/InsertRoundTripCase.cs | Adds live-server insert round-trip cases for Nested(...) under flatten_nested=0. |
| ClickHouse.Driver.Tcp.Tests/Types/NullableColumnCodecTests.cs | Updates an “unsupported inner type” test case now that Nested(...) is supported. |
| ClickHouse.Driver.Tcp.Tests/Types/NestedColumnCodecTests.cs | New unit test suite for Nested codec: round-trips, byte identity, slicing, error cases, and >7 fields. |
| ClickHouse.Driver.Tcp.Tests/Types/ColumnCodecRegistryTests.cs | Updates registry “unsupported but well-formed type” test now that Nested(...) is supported. |
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 21, 2026 17:02
fb22255 to
9b9a653
Compare
alex-clickhouse
changed the title
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 22, 2026 13:23
9b9a653 to
8bb6b78
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 22, 2026 15:26
8bb6b78 to
221d8bd
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 22, 2026 16:22
221d8bd to
1d7a531
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 23, 2026 08:53
1d7a531 to
bf4745a
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 23, 2026 09:01
bf4745a to
e40ccfd
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 23, 2026 11:47
e40ccfd to
260a347
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 23, 2026 18:04
260a347 to
1fafd4e
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 28, 2026 15:26
babb5a2 to
c5931ac
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
2 times, most recently
from
July 28, 2026 18:59
f4d3798 to
f5e46a9
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 28, 2026 19:18
f5e46a9 to
9e4a54f
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 29, 2026 07:05
9e4a54f to
fafac87
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 29, 2026 13:26
fafac87 to
bd81e39
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 29, 2026 15:47
bd81e39 to
538f6c4
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 30, 2026 08:37
538f6c4 to
1d64828
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 30, 2026 09:10
1d64828 to
f3f8dfd
Compare
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 30, 2026 09:47
f3f8dfd to
03f1622
Compare
A Nested(name1 T1, ..., namen Tn) column carried as a single wire column
(the flatten_nested = 0 form) is laid out byte-identically to
Array(Tuple(T1, ..., Tn)): the fields' state-prefix phases, then a per-row
offsets stream, then each field's flattened element stream. Under the
server-default flatten_nested = 1 there is no Nested wire type — the server
presents N parallel dotted Array(T_i) columns, already handled as plain
arrays — so only the single-column form needs a codec.
Rather than reuse the Array(Tuple(...)) codecs (which would inherit the
tuple's 7-element cap, a limit a real Nested routinely exceeds, and expose
named fields as positional .ItemN access), Nested gets a dedicated,
arity-agnostic columnar representation:
- NestedColumn : IColumn<object[][]> keeps one flat field column per field
plus a shared per-row offsets array (the wire's own shape). The fast path
is columnar and allocation-free: GetField(index/name) returns a field's
flat column, Offsets maps a row to its element range, FieldNames lists the
fields. The IColumn<object[][]> surface materializes each row as an
array-of-records (boxed) as a convenience for generic consumers.
- NestedColumnCodec owns one child codec per field and loops them for every
phase; it reads the offsets and each field stream into a NestedColumn and
writes the sliced offsets plus each sliced field stream. The dense
NestedColumn is the only write source — there is deliberately no
row-oriented insert form, which would need a per-row record type and
reintroduce an arity cap. Fixed-width fields are priced O(1) by the insert
splitter; only variable-width fields are walked per element.
Fields must be named (an unnamed field is rejected). Nested composes inside a
field (Nested(a Nullable(T), b Array(T), c Tuple(...), ...)); the server
rejects Nullable(Nested(...)), so — like Array/Tuple/Map — nullability
composes inside a field. Two registry tests that used Nested as their stand-in
"unsupported type" now use LowCardinality(String), which remains unsupported.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A Nested column is only ever the dense NestedColumn, so Densify recurses each field codec's own Densify over its field column — turning e.g. a Nested with an Array or Nullable field dense all the way down. An already-dense Nested (every field dense) is returned by reference. When only some fields need densifying, the rebuilt NestedColumn keeps the unchanged fields by reference (still owned by the source column) and builds the changed ones fresh. It previously borrowed all of them (ownsFields: false), so disposing the wrapper freed nothing and the freshly built fields leaked. NestedColumn now takes a per-field ownership mask (RestrictOwnership), and the rebuild flags exactly the fields it created, so disposal frees those without double-disposing the borrowed ones. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nested writes its shared offsets then each field's flattened run straight from the ergonomic source, recursing into the field codecs, with no densify pre-pass or byte measurement. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR 455 (round 2): Materialize indexed the pooled offsets array directly, which can be longer than RowCount + 1, so an out-of-range row returned stale offsets instead of throwing. Index through the RowCount-sliced Offsets span, matching the bounds behavior of the other columns. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This epic wrote its integration cases alongside its unit tests, so the overlap is small -- one deletion and two trims rather than a sweep. Deleted ReadColumn_MoreThanSevenFields_RoundTrips: the "Nested(8 fields) [uncapped]" case has the same arity and offsets, and the test asserted only FieldCount/RowCount/field values, all of which the integration comparison reproduces. Trimmed the row-value assertions from ReadColumn_WriteThenRead_FixedAndStringFieldsRoundTripWithEmptyRows; it stays for TypeName, FieldNames and Offsets, which the integration comparison never inspects, but the boxed array-of-records values are covered by the "Nested(a UInt8, b String)" case. Trimmed ReadColumn_CompositeFields_RoundTrip to its Tuple field. The Nullable and Array field compositions are covered by the "Nested(a Nullable(Int32), b Array(String))" case; a Tuple field has no integration case, so that assertion is the only reason the test remains. Values_MaterializesEveryRowAsArrayOfRecords is untouched -- it is the sole cover for NestedColumn's lazily materialized Values cache, which GetValue never reaches. Verified against ClickHouse 26.6 -- 870 tests pass, integration included. Co-Authored-By: Claude <noreply@anthropic.com>
Add INestedColumn, the public read surface for a Nested(...) carried as one wire column: the flat per-field columns, the offsets array they all share, and the field names. For Nested this view is the primary access path rather than an optimization. A Nested can carry any number of fields, so there is no generic per-row value type for it and the IColumn<T> surface has to degrade to object[][] — a boxed object[] per record, per row. The field columns were already public members, but on an internal class, so nothing outside the assembly could reach them. Offsets stays sliced to RowCount + 1, so the public span cannot expose the tail of the pooled buffer behind it.
Drop the unqualified "zero-copy" (reading a field column's values still materializes for string-like and composite field types), say plainly that a field column is the block's to dispose rather than the caller's, and document the IndexOutOfRangeException from GetField(int) — the string overload already documented its KeyNotFoundException while the int overload documented nothing.
Same split as Array and Tuple. Co-Authored-By: Claude <noreply@anthropic.com>
alex-clickhouse
force-pushed
the
tcp/epic-i5-nested
branch
from
July 31, 2026 16:14
03f1622 to
3535a16
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
Nested(...)support (I5) for the TCP/Native client, stacked on top of FixedString (#452).What
Nested(name1 T1, …)carried as a single wire column (theflatten_nested = 0form) is laid out byte-identically toArray(Tuple(T1, …)). Under the server-defaultflatten_nested = 1there is noNestedwire type — the server presents N parallel dottedArray(T_i)columns, already handled as plain arrays — so only the single-column form needs a codec.Design: dedicated columnar representation (not
Array(Tuple)reuse)Reusing the Array+Tuple codecs would inherit the tuple's 7-element cap — a limit a real
Nestedroutinely exceeds — and expose named fields as positional.ItemNaccess. Instead:NestedColumn : IColumn<object[][]>— one flat field column per field + a shared per-row offsets array (the wire's own shape). Fast path is columnar and allocation-free:GetField(index/name),Offsets,FieldNames. TheIColumn<object[][]>surface (each row = array-of-records, boxed) is a convenience for generic consumers.NestedColumnCodec— owns one child codec per field; reads offsets + each field stream into aNestedColumn; writes sliced offsets + each sliced field stream. Fixed-width fields priced O(1) by the insert splitter. The denseNestedColumnis the only write source (no row-oriented insert form — that would reintroduce an arity cap).Fields must be named. Nested composes inside a field (
Nested(a Nullable(T), b Array(T), c Tuple(...))); the server rejectsNullable(Nested(...)), so nullability composes inside a field, like Array/Tuple/Map.The full options analysis (A
Array(Tuple)/ValueTuple · B wide tuples · C chosen · Dobject[]) is captured in the localdocs/design-decisions.md(D4, Nested note).Tests
NestedColumnCodecTests): round-trip incl. empty rows, byte-identity vsArray(Tuple), 8-field uncapped, composite fields (nullable/array/tuple), columnar field access, sliced write,MeasureRowBytes(fixed + variable),CanWrite, constructor guards, truncated-stream / offset-overflow / non-monotonic, error cases.Nested(a UInt8, b String), nullable+array fields, and an 8-field case.Array.MaxLengthoffset-buffer guard (untestable; identical to the Array/Map siblings).A pooled-buffer double-return bug (offsets returned to
ArrayPooltwice on a mid-read failure) was found by the truncated-stream test and fixed.🤖 Generated with Claude Code