TCP H2: Add FixedString(N) support - #452
Conversation
TriageCategory: Summary What this impacts
Concerns
Required reviewer action
|
There was a problem hiding this comment.
Pull request overview
This PR adds full TCP (native) client support for ClickHouse FixedString(N) by introducing a fixed-width binary column representation and codec, registering it in the TCP codec registry, and extending the TCP test suite with unit + live round-trip coverage.
Changes:
- Added
FixedStringColumnto exposeFixedString(N)rows as fixed-stride slices over a pooled blob (with lazy per-rowbyte[]materialization). - Added
FixedStringColumnCodecto parseN, bulk-read/writerowCount * Nbytes, right-pad short values toN, and reject over-length/null rows (nulls only viaNullable+ placeholder). - Extended TCP tests with
FixedStringunit tests and integration round-trip cases (includingNullable(...)andArray(...)compositions).
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| ClickHouse.Driver.Tcp/Types/FixedStringColumn.cs | New fixed-stride pooled-blob column implementation with zero-copy GetBytes and cached byte[] view. |
| ClickHouse.Driver.Tcp/Types/ColumnCodecRegistry.cs | Registers FixedString as a parameterized codec factory. |
| ClickHouse.Driver.Tcp/Types/Codecs/FixedStringColumnCodec.cs | New codec handling parsing, bulk read, fixed-width write with padding, and null placeholder semantics. |
| ClickHouse.Driver.Tcp.Tests/Utilities/InsertRoundTripCase.cs | Adds live-server insert/read-back cases for FixedString, Nullable(FixedString), and Array(FixedString). |
| ClickHouse.Driver.Tcp.Tests/Types/FixedStringColumnCodecTests.cs | New unit tests covering parse errors, padding, multi-chunk padding, null/over-length rejection, stride correctness, and bounds checks. |
5b44542 to
88acb7c
Compare
5e11626 to
c6f2f8f
Compare
c6f2f8f to
a9d5c00
Compare
a9d5c00 to
f900c28
Compare
f900c28 to
cceb343
Compare
cceb343 to
0a580b4
Compare
0a580b4 to
eba2756
Compare
eba2756 to
49e88c3
Compare
49e88c3 to
4cb8601
Compare
a113936 to
2ebf264
Compare
2ebf264 to
f6b06bf
Compare
f6b06bf to
282ea17
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
282ea17 to
4c34975
Compare
4c34975 to
50712eb
Compare
50712eb to
12ef7f1
Compare
12ef7f1 to
d765302
Compare
d765302 to
4e1f278
Compare
4e1f278 to
06a3d17
Compare
53618b5 to
4d5e431
Compare
FixedString(N) is N contiguous bytes per row with no length prefix, so it is a fixed-width type: the codec reports FixedRowByteSize = N and the insert splitter prices it in O(1). Rows are read in one bulk transfer into a pooled blob at a fixed stride and surfaced as a per-row byte[] via a bespoke FixedStringColumn (zero-copy GetBytes, GetString(encoding), lazy materialized view) — mirroring StringColumn but without an offsets array. byte[] is the honest surface for a byte-oriented type, so embedded NULs and non-UTF-8 bytes round-trip intact. On write, a value is emitted verbatim and right-padded with zero bytes to N; over-length and null rows are rejected (nulls only reach the wire via Nullable, which substitutes the empty-array placeholder). Nullable( FixedString(N)) composes through the reference-nullable shape. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FixedString writes each row's fixed-width value straight from the ergonomic source via the value-writer path, with no byte measurement. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR 452 feedback: WriteColumn read every row through the IColumn<byte[]> indexer, which for a dense FixedStringColumn allocates a byte[] per row (GetBytes(row).ToArray()) — the hot path when re-inserting a value read straight back. Special-case FixedStringColumn to write directly from its zero-copy GetBytes span via a new span-based WriteRow overload; scattered views still fall back to the indexer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR 452 (round 2): the span-based WriteRow overload used a "value" string literal for the paramName; use nameof(value) to match the byte[] overload and stay correct across renames. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
4d5e431 to
4b77092
Compare
The write path used to right-pad a short byte[] to N zero bytes. Drop that: a wrong-width value now throws. Padding silently rewrites the caller's data and hides whatever produced the wrong width, and the HTTP path already rejects it — FixedStringType.WriteByteArray requires a byte[] to be exactly N bytes (as do its ReadOnlyMemory and Stream overloads), padding only for string. So the same package was accepting over TCP what it refused over HTTP. Padding belongs with the deferred string-write overload, not with byte[]. Separately, write a dense FixedStringColumn as one contiguous blit instead of walking it row by row. Its rows already sit at the wire stride, so rows [start, start + length) are a single slice of the blob — the hot path when re-inserting a value read straight back. This is independent of the padding change: no other write source is contiguous, since user data arrives as jagged byte[][]. The blit is guarded on the column's width matching the codec's. That guard fixes a latent bug rather than merely enabling the fast path: the old dense branch had no width check at all, and CanWrite only tests the CLR element type, so inserting a FixedString(2) read-back into a FixedString(4) target silently zero-padded every row to 4 bytes. Also: - NullPlaceholder is now N zero bytes rather than the empty array it relied on the pad loop to widen. Built lazily: a codec is resolved per column per block, so allocating it eagerly would charge every read of a wide FixedString for a buffer only the Nullable write path touches. - Both rejection messages carry the offending position — its row in a column, its index within the row's array under Array(FixedString(N)) — since rejecting is now the only signal a caller gets. - FixedStringColumn gains Size and a GetBytes(start, length) range accessor, bounded against RowCount so an over-long range cannot blit a stale region of the pooled blob. Tests: the three width rejections collapse into one parametrized case; the FixedString(6) padding round-trip is gone (the behavior it pinned no longer exists) and FixedString(200) replaces it, keeping the >64-byte width coverage at the layer that owns per-type values and giving the blit a stride wider than one row. New unit tests cover the sub-range blit, the mismatched-width fallback, range bounds past RowCount, and the placeholder — all shapes no server round-trip can reach. Adds a Tuple(FixedString(4), String) case for the one entrance that reaches the strict per-value branch through a field projection. Co-Authored-By: Claude <noreply@anthropic.com>
What it does
FixedString(N)is the last Epic H type: N contiguous bytes per row, no length prefix, so a column body is exactlynum_rows * Nbytes.FixedStringColumnsurfaces each row as abyte[]. It mirrorsStringColumn's design (pooled blob, zero-copyGetBytes(row),GetString(row, encoding), lazy materialized view) but uses a fixed stride instead of an offsets array.byte[]is the honest default for a byte-oriented type, so embedded NULs and non-UTF-8 bytes round-trip intact. Out-of-range access is bounded againstRowCount(not the rented blob) so it fails fast rather than returning stale pooled data.FixedStringColumnCodecparsesN(rejecting missing/non-integer/non-positive/multi-arg), bulk-reads the body, and on write emits each value verbatim.Nullable(FixedString(N))composes through the existing reference-nullable shape, substituting an N-zero-byte placeholder so a null never reaches this codec.Write semantics: exactly N bytes, no padding
A value must be exactly N bytes. Over-length is rejected (matching the server, which errors rather than truncating) and so is under-length — the write path does not zero-pad.
Two reasons:
ClickHouse.Driver/Types/FixedStringType.csrequires abyte[]to be exactly N bytes (as do itsReadOnlyMemory<byte>andStreamoverloads) and pads only forstring. Padding here would mean the same package accepts over TCP what it refuses over HTTP.That also draws the line for the deferred string-write overload: padding is a
stringaffordance, not abyte[]one, and belongs there when it lands — mirroringFixedStringType.WriteString.Both rejection messages carry the offending position — its row in a column, its index within the row's array under
Array(FixedString(N))— since rejecting is now the caller's only signal.Dense write is one blit
A dense
FixedStringColumnalready holds its rows at the wire stride, so rows[start, start + length)are a single contiguous slice of the blob and go out in oneWriteBytes— the hot path when re-inserting a value read straight back, and the same shapeFixedWidthColumnCodecuses viaISpanColumn<T>. Every other write source is jaggedbyte[][](a caller'sArrayColumn<byte[]>, aNullablesubstitute, aTuplefield, anArrayelement run), so those keep the per-row path.The blit is guarded on the column's width matching the codec's. That guard fixes a latent bug, not just the fast path: the previous dense branch had no width check at all, and
CanWriteonly tests the CLR element type (column is IColumn<byte[]>), so inserting aFixedString(2)read-back into aFixedString(4)target silently zero-padded every row to 4 bytes. A mismatched column now falls through to the per-row path and is rejected on width.Testing
FixedStringColumnCodecTests— parse errors, exact-width write, wrong-width rejection (parametrized: empty / short / long), null-row rejection, embedded-NUL and non-UTF-8 round-trip, zero rows, out-of-range before and after cache materialization,CanWrite, plus four shapes no server round-trip can reach: the sub-range blit, the mismatched-width dense fallback,GetBytes(start, length)bounds pastRowCount, and the placeholder's content.InsertRoundTripCase— live-server round-trips forFixedString(4),FixedString(200)(a stride wider than one row, so a slipped blit is visible),Nullable(FixedString(4))(interleaved + all-null, per the "always test insideNullable" rule),Array(FixedString(4)), andTuple(FixedString(4), String)— the one entrance that reaches the strict per-value branch through a field projection rather than a dense blob.catch, matching sibling codecs.Notes
checkedmultiply already prevents overflow-to-negative; a policy cap on blob size is deferred by design decision Q5, so none is introduced here.NullPlaceholderis N zero bytes, built lazily — a codec is resolved per column per block, so allocating it eagerly would charge every read of a wideFixedStringfor a buffer only theNullablewrite path touches.byte[]only (notstring) for now, keepingNullablecomposition single-write-type; string-write ergonomics can be a follow-up.🤖 Generated with Claude Code