Skip to content

TCP H2: Add FixedString(N) support - #452

Draft
alex-clickhouse wants to merge 5 commits into
tcp/epic-i4-mapfrom
tcp/epic-h2-fixedstring
Draft

TCP H2: Add FixedString(N) support#452
alex-clickhouse wants to merge 5 commits into
tcp/epic-i4-mapfrom
tcp/epic-h2-fixedstring

Conversation

@alex-clickhouse

@alex-clickhouse alex-clickhouse commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

What it does

FixedString(N) is the last Epic H type: N contiguous bytes per row, no length prefix, so a column body is exactly num_rows * N bytes.

  • FixedStringColumn surfaces each row as a byte[]. It mirrors StringColumn's design (pooled blob, zero-copy GetBytes(row), GetString(row, encoding), lazy materialized view) but uses a fixed stride instead of an offsets array. byte[] is the honest default for a byte-oriented type, so embedded NULs and non-UTF-8 bytes round-trip intact. Out-of-range access is bounded against RowCount (not the rented blob) so it fails fast rather than returning stale pooled data.
  • FixedStringColumnCodec parses N (rejecting missing/non-integer/non-positive/multi-arg), bulk-reads the body, and on write emits each value verbatim. Nullable(FixedString(N)) composes through the existing reference-nullable shape, substituting an N-zero-byte placeholder so a null never reaches this codec.

Write semantics: exactly N bytes, no padding

A value must be exactly N bytes. Over-length is rejected (matching the server, which errors rather than truncating) and so is under-length — the write path does not zero-pad.

Two reasons:

  1. Padding silently rewrites the caller's data, so a wrong-width value reaches the server looking correct instead of surfacing the bug that produced it.
  2. The HTTP path already rejects it. ClickHouse.Driver/Types/FixedStringType.cs requires a byte[] to be exactly N bytes (as do its ReadOnlyMemory<byte> and Stream overloads) and pads only for string. Padding here would mean the same package accepts over TCP what it refuses over HTTP.

That also draws the line for the deferred string-write overload: padding is a string affordance, not a byte[] one, and belongs there when it lands — mirroring FixedStringType.WriteString.

Both rejection messages carry the offending position — its row in a column, its index within the row's array under Array(FixedString(N)) — since rejecting is now the caller's only signal.

Dense write is one blit

A dense FixedStringColumn already holds its rows at the wire stride, so rows [start, start + length) are a single contiguous slice of the blob and go out in one WriteBytes — the hot path when re-inserting a value read straight back, and the same shape FixedWidthColumnCodec uses via ISpanColumn<T>. Every other write source is jagged byte[][] (a caller's ArrayColumn<byte[]>, a Nullable substitute, a Tuple field, an Array element run), so those keep the per-row path.

The blit is guarded on the column's width matching the codec's. That guard fixes a latent bug, not just the fast path: the previous dense branch had no width check at all, and CanWrite only tests the CLR element type (column is IColumn<byte[]>), so inserting a FixedString(2) read-back into a FixedString(4) target silently zero-padded every row to 4 bytes. A mismatched column now falls through to the per-row path and is rejected on width.

Testing

  • FixedStringColumnCodecTests — parse errors, exact-width write, wrong-width rejection (parametrized: empty / short / long), null-row rejection, embedded-NUL and non-UTF-8 round-trip, zero rows, out-of-range before and after cache materialization, CanWrite, plus four shapes no server round-trip can reach: the sub-range blit, the mismatched-width dense fallback, GetBytes(start, length) bounds past RowCount, and the placeholder's content.
  • InsertRoundTripCase — live-server round-trips for FixedString(4), FixedString(200) (a stride wider than one row, so a slipped blit is visible), Nullable(FixedString(4)) (interleaved + all-null, per the "always test inside Nullable" rule), Array(FixedString(4)), and Tuple(FixedString(4), String) — the one entrance that reaches the strict per-value branch through a field projection rather than a dense blob.
  • Full TCP suite green (865 tests, incl. containerized round-trips). Both new files fully covered except the defensive read-failure leak-guard catch, matching sibling codecs.

Notes

  • No size cap on N: the checked multiply already prevents overflow-to-negative; a policy cap on blob size is deferred by design decision Q5, so none is introduced here.
  • NullPlaceholder is N zero bytes, built lazily — a codec is resolved per column per block, so allocating it eagerly would charge every read of a wide FixedString for a buffer only the Nullable write path touches.
  • Write accepts byte[] only (not string) for now, keeping Nullable composition single-write-type; string-write ergonomics can be a follow-up.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown

Triage

Category: featureRisk: high

Summary
Adds FixedString(N) support to the TCP driver (ClickHouse.Driver.Tcp). The PR introduces FixedStringColumn (a fixed-stride pooled-blob column surfacing byte[] per row) and FixedStringColumnCodec (parses the N argument, bulk-reads rowCount × N bytes on read, and emits verbatim bytes with zero-right-padding on write). The codec is registered in ColumnCodecRegistry via a new factory entry. This is the last "Epic H" type for the TCP driver. Tests cover parse-error paths, padding, over-length/null rejection, embedded-NUL/non-UTF-8 round-trips, zero-row and out-of-range bounds, Nullable(FixedString(N)), and Array(FixedString(N)) live server round-trips.

What this impacts

  • ClickHouse.Driver.Tcp/Types/ — two new files: FixedStringColumn.cs and Codecs/FixedStringColumnCodec.cs
  • ClickHouse.Driver.Tcp/Types/ColumnCodecRegistry.cs — new AddFactory("FixedString", …) entry wires in type-argument parsing
  • Binary read/write path: fixed-stride bulk read (ReadBytesAsync) and zero-padded write loop with stackalloc zero buffer
  • InsertRoundTripCase extended for bare, padding, Nullable, and Array variants

Concerns

  • High rule: binary protocol + type system — new binary read/write serialization path fires both the "Binary protocol / serialization layout" and "Type system / binary read/write paths" rules regardless of the additive-only nature of the change.
  • Thread-safety assumption in FixedStringColumn.Values — the lazy cache initialization at line ~82 of FixedStringColumn.cs has no synchronization guard. The comment says "Single-consumer per connection, so the lazy fill needs no synchronization." If the assumption is ever violated (e.g., two threads calling Values on the same column object), the cache could be double-allocated or partially visible. Reviewer should verify this assumption is enforced at the call site, not just documented.
  • No size cap on N — acknowledged in the PR body as a deferred policy decision (Q5). The checked multiply prevents integer overflow, but a caller can request a single-row read of 2 GB; reviewer should confirm this is acceptable for now.
  • PR is in DRAFT state — may still be in flux.

Required reviewer action

  • PR body already includes an architectural description. Human reviewer should verify binary encoding correctness (exact N-byte stride, null-placeholder → N zeros, padding contract) against ClickHouse protocol docs, and confirm the single-consumer thread-safety assumption for Values.

@alex-clickhouse
alex-clickhouse requested a review from Copilot July 21, 2026 09:19
@alex-clickhouse alex-clickhouse changed the title Add FixedString(N) support for the TCP client TCP H2: Add FixedString(N) support Jul 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds full TCP (native) client support for ClickHouse FixedString(N) by introducing a fixed-width binary column representation and codec, registering it in the TCP codec registry, and extending the TCP test suite with unit + live round-trip coverage.

Changes:

  • Added FixedStringColumn to expose FixedString(N) rows as fixed-stride slices over a pooled blob (with lazy per-row byte[] materialization).
  • Added FixedStringColumnCodec to parse N, bulk-read/write rowCount * N bytes, right-pad short values to N, and reject over-length/null rows (nulls only via Nullable + placeholder).
  • Extended TCP tests with FixedString unit tests and integration round-trip cases (including Nullable(...) and Array(...) compositions).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
ClickHouse.Driver.Tcp/Types/FixedStringColumn.cs New fixed-stride pooled-blob column implementation with zero-copy GetBytes and cached byte[] view.
ClickHouse.Driver.Tcp/Types/ColumnCodecRegistry.cs Registers FixedString as a parameterized codec factory.
ClickHouse.Driver.Tcp/Types/Codecs/FixedStringColumnCodec.cs New codec handling parsing, bulk read, fixed-width write with padding, and null placeholder semantics.
ClickHouse.Driver.Tcp.Tests/Utilities/InsertRoundTripCase.cs Adds live-server insert/read-back cases for FixedString, Nullable(FixedString), and Array(FixedString).
ClickHouse.Driver.Tcp.Tests/Types/FixedStringColumnCodecTests.cs New unit tests covering parse errors, padding, multi-chunk padding, null/over-length rejection, stride correctness, and bounds checks.

@alex-clickhouse
alex-clickhouse marked this pull request as draft July 21, 2026 09:27
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from 5b44542 to 88acb7c Compare July 21, 2026 11:03
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch 2 times, most recently from 5e11626 to c6f2f8f Compare July 22, 2026 15:26
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from c6f2f8f to a9d5c00 Compare July 22, 2026 16:22
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from a9d5c00 to f900c28 Compare July 23, 2026 08:52
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from f900c28 to cceb343 Compare July 23, 2026 09:00
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from cceb343 to 0a580b4 Compare July 23, 2026 11:47
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from 0a580b4 to eba2756 Compare July 23, 2026 18:04
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from eba2756 to 49e88c3 Compare July 24, 2026 07:19
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from 49e88c3 to 4cb8601 Compare July 24, 2026 07:26
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch 2 times, most recently from a113936 to 2ebf264 Compare July 28, 2026 15:26
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from 2ebf264 to f6b06bf Compare July 28, 2026 16:01
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from f6b06bf to 282ea17 Compare July 28, 2026 18:59
@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from 282ea17 to 4c34975 Compare July 28, 2026 19:18
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from 4c34975 to 50712eb Compare July 29, 2026 07:05
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from 50712eb to 12ef7f1 Compare July 29, 2026 13:26
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from 12ef7f1 to d765302 Compare July 29, 2026 15:47
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from d765302 to 4e1f278 Compare July 30, 2026 08:37
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from 4e1f278 to 06a3d17 Compare July 30, 2026 09:10
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch 2 times, most recently from 53618b5 to 4d5e431 Compare July 31, 2026 16:14
alex-clickhouse and others added 4 commits August 4, 2026 17:12
FixedString(N) is N contiguous bytes per row with no length prefix, so
it is a fixed-width type: the codec reports FixedRowByteSize = N and the
insert splitter prices it in O(1). Rows are read in one bulk transfer
into a pooled blob at a fixed stride and surfaced as a per-row byte[]
via a bespoke FixedStringColumn (zero-copy GetBytes, GetString(encoding),
lazy materialized view) — mirroring StringColumn but without an offsets
array. byte[] is the honest surface for a byte-oriented type, so embedded
NULs and non-UTF-8 bytes round-trip intact.

On write, a value is emitted verbatim and right-padded with zero bytes to
N; over-length and null rows are rejected (nulls only reach the wire via
Nullable, which substitutes the empty-array placeholder). Nullable(
FixedString(N)) composes through the reference-nullable shape.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FixedString writes each row's fixed-width value straight from the
ergonomic source via the value-writer path, with no byte measurement.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR 452 feedback: WriteColumn read every row through the IColumn<byte[]>
indexer, which for a dense FixedStringColumn allocates a byte[] per row
(GetBytes(row).ToArray()) — the hot path when re-inserting a value read
straight back. Special-case FixedStringColumn to write directly from its
zero-copy GetBytes span via a new span-based WriteRow overload; scattered
views still fall back to the indexer.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR 452 (round 2): the span-based WriteRow overload used a "value" string
literal for the paramName; use nameof(value) to match the byte[] overload
and stay correct across renames.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-h2-fixedstring branch from 4d5e431 to 4b77092 Compare August 4, 2026 15:16
The write path used to right-pad a short byte[] to N zero bytes. Drop
that: a wrong-width value now throws. Padding silently rewrites the
caller's data and hides whatever produced the wrong width, and the HTTP
path already rejects it — FixedStringType.WriteByteArray requires a
byte[] to be exactly N bytes (as do its ReadOnlyMemory and Stream
overloads), padding only for string. So the same package was accepting
over TCP what it refused over HTTP. Padding belongs with the deferred
string-write overload, not with byte[].

Separately, write a dense FixedStringColumn as one contiguous blit
instead of walking it row by row. Its rows already sit at the wire
stride, so rows [start, start + length) are a single slice of the blob —
the hot path when re-inserting a value read straight back. This is
independent of the padding change: no other write source is contiguous,
since user data arrives as jagged byte[][].

The blit is guarded on the column's width matching the codec's. That
guard fixes a latent bug rather than merely enabling the fast path: the
old dense branch had no width check at all, and CanWrite only tests the
CLR element type, so inserting a FixedString(2) read-back into a
FixedString(4) target silently zero-padded every row to 4 bytes.

Also:
- NullPlaceholder is now N zero bytes rather than the empty array it
  relied on the pad loop to widen. Built lazily: a codec is resolved per
  column per block, so allocating it eagerly would charge every read of
  a wide FixedString for a buffer only the Nullable write path touches.
- Both rejection messages carry the offending position — its row in a
  column, its index within the row's array under Array(FixedString(N)) —
  since rejecting is now the only signal a caller gets.
- FixedStringColumn gains Size and a GetBytes(start, length) range
  accessor, bounded against RowCount so an over-long range cannot blit a
  stale region of the pooled blob.

Tests: the three width rejections collapse into one parametrized case;
the FixedString(6) padding round-trip is gone (the behavior it pinned no
longer exists) and FixedString(200) replaces it, keeping the >64-byte
width coverage at the layer that owns per-type values and giving the
blit a stride wider than one row. New unit tests cover the sub-range
blit, the mismatched-width fallback, range bounds past RowCount, and the
placeholder — all shapes no server round-trip can reach. Adds a
Tuple(FixedString(4), String) case for the one entrance that reaches the
strict per-value branch through a field projection.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants