TCP J1/J2: Add LowCardinality(T) and LowCardinality(Nullable(T)) support by alex-clickhouse · Pull Request #456 · ClickHouse/clickhouse-cs

alex-clickhouse · 2026-07-22T08:00:24Z

What

Implements LowCardinality(T) for the TCP/Native client (Epic J1), for both non-nullable and nullable inner types. A LowCardinality column replaces a run of inner values with a block-local dictionary of the distinct values plus one key per row indexing into it.

Works both top-level and under composites (Array(LowCardinality(String)) verified).

Design

Wire body: metadata UInt64 (key-width code in the low byte + HasAdditionalKeys | NeedUpdateDictionary = 0x600), dict_size, dictionary values via the inner codec, keys_count, then keys at 1 << code bytes. Key width is the smallest that indexes the dictionary (UInt8/16/32). An empty (or empty-flattened) column writes no body.
Column surface: dense LowCardinalityColumn<T> (dictionary column + int[] keys) reconstructs dict[keys[row]] and is the zero-copy write source; the ergonomic write source is a plain IColumn<T>, deduplicated into a fresh block-local dictionary on write. Follows the existing Array/Nullable shape-bridge pattern (ILowCardinalityShape + reflection-cached per-element-type shape). Each Native block ships a self-contained dictionary, so the codec stays a shared singleton.
byte[] dedup: FixedString element type (byte[]) uses a structural comparer so equal values collapse to one dictionary slot rather than defaulting to reference equality.
Hostile-stream validation: unknown key-width code, global-dictionary bit, missing additional-keys bit, dict_size/key-stream overflow, keys_count mismatch, and out-of-range keys are all rejected.

`LowCardinality(Nullable(T))`

The composability gap the non-nullable path left open: the codec's single inner field was serving as both the dictionary's wire serializer and the element-type/null authority. Nullable inner forces those apart.

The dictionary stays bare T — no null-map. Nullability is expressed positionally by two reserved leading slots: dict[0] is the NULL marker and dict[1] the inner default, so real distinct values start at dict[2]. Every NULL row's key points at dict[0].
Resolve the bare inner, not the Nullable codec. Create detects a Nullable inner and resolves the codec for the unwrapped type (registry.ResolveNode(innerNode.Arguments[0])), carrying a nullable flag. Resolving the Nullable codec would frame a null-map inside the dictionary stream and desynchronize the reader.
Surface type diverges from the dictionary type. For a value inner, the dictionary decodes as IColumn<T> but the column surfaces T? (NullableLowCardinalityValueColumn<T>); for a reference inner it surfaces the nullable reference (NullableLowCardinalityReferenceColumn<T>). Both read a key == 0 row back as NULL. Selected via a value/reference shape split mirroring Nullable's.
Write dedup, inverted default folding. A NULL row maps to key 0; a present value equal to the inner default reuses the reserved default slot 1 (so it round-trips as a present default, distinct from NULL) rather than folding into slot 0 as the non-nullable path does.
Dense re-emit safety. The dense nullable columns are the zero-copy re-emit source via a new IDenseLowCardinality<T> marker. The non-nullable LowCardinalityColumn<T> deliberately does not implement it — its single reserved slot would be misread as NULL if re-emitted verbatim under a nullable codec.

Like Array/Tuple/Map/Nested, the server rejects Nullable(LowCardinality(T)), so nullability composes inside as LowCardinality(Nullable(T)).

Testing

Codec unit tests cover the documented wire bytes, key-width promotion, hostile streams, dense re-emit, and the nullable-specific paths (present-default-vs-NULL, value/reference/FixedString, empty slice). Verified end-to-end against ClickHouse 26.6: LowCardinality(T) and LowCardinality(Nullable(T)) round-trips for String, UInt32, and FixedString(4), interleaving NULLs with present values (including a present inner-default value). New/changed codec files are ~98% line / 100% method covered.

github-actions · 2026-07-22T08:02:14Z

Triage

Category: feature • Risk: high

Summary
This PR implements LowCardinality(T) and LowCardinality(Nullable(T)) column codecs for the TCP/Native client (Epic J1). The wire format uses a block-local dictionary of distinct values plus per-row integer keys; key width auto-promotes from UInt8→UInt16→UInt32 as the dictionary grows. The nullable variant reserves two leading dictionary slots (slot 0 = NULL, slot 1 = inner default) rather than using a null-map, requiring a careful split between the "dictionary inner codec" (bare T) and the "surface type" (T?). Changes touch ClickHouse.Driver.Tcp/Types/Codecs/ (new LowCardinalityColumnCodec, ILowCardinalityShape, three new column classes), a comment-only update to BlockReader.cs (removing the outstanding TODO), and ~570 lines of new codec unit tests plus ~76 lines of round-trip integration cases.

What this impacts

Binary protocol / TCP type system — new read and write paths for LowCardinality in the Native binary format; incorrect framing or slot indexing corrupts every column in a block.
LowCardinalityColumn<T> / NullableLowCardinalityReferenceColumn<T> / NullableLowCardinalityValueColumn<T> — new pooled-array columns with their own Dispose paths; ArrayPool misuse would silently corrupt later reads.
Composite types — Array(LowCardinality(...)) flattens through the new codec; the nullable-inside-array case (Array(LowCardinality(Nullable(T)))) has a separate round-trip case, but any composites beyond Array are untested.
ColumnCodecRegistryTests — the "unsupported type" sentinel was changed from LowCardinality(String) to Variant(...), which is correct but a silent weakening if Variant ever becomes supported before the test is updated.

Concerns

High-risk rule fired — binary protocol / type system: New binary read and write paths for a dictionary-bearing type with reserved slot semantics; an off-by-one in the nullable reserved slots (0 = NULL, 1 = default) or in key-width promotion boundary arithmetic (< byte.MaxValue vs. ≤) would produce data corruption that round-trip tests may not catch if the server is lenient.
High-risk rule fired — pooled arrays and Dispose: Three new column classes rent from ArrayPool; correctness of pooling, bounds guards (rowCount vs. key-array length), and Dispose safety under composite reads all warrant careful inspection.
Draft PR: The PR is currently in draft state; the risk assessment applies to the diff as-is, but it should not proceed to merge review until the author marks it ready.
Reflection-cached shapes: The ILowCardinalityShape per-element-type cache uses reflection to construct typed column instances; reviewers should verify the cache is thread-safe and that the reflection path cannot throw on normal types.
BlockReader.cs comment-only change: The removed TODO stated that LowCardinality must emit a state prefix regardless of row count. The updated comment says it emits the prefix only for non-zero row count. Reviewers should confirm this matches the ClickHouse server's expectation for zero-row schema headers.

Required reviewer action

PR body must include an architectural description before review — it does (design section covers wire body, column surface, dedup, and nullable-specific paths), so the architectural requirement is satisfied; at least one human reviewer with binary-protocol familiarity is needed before merge.

Copilot

Pull request overview

Adds TCP/Native (Epic J1) support for ClickHouse LowCardinality(T) columns (non-nullable inner types), including a dense column representation and a non-generic codec that delegates typed work to cached per-element “shape” bridges. This integrates LowCardinality into the TCP codec registry and validates hostile/invalid streams while providing round-trip and byte-level codec tests.

Changes:

Introduces LowCardinalityColumnCodec + wire helpers and a dense LowCardinalityColumn<T> representation.
Adds a per-element-type shape bridge (ILowCardinalityShape, LowCardinalityShape<T>, cached via LowCardinalityShapes) to enable generic write/read logic from a non-generic codec pipeline.
Expands TCP tests to cover round-trips, exact byte layout, key-width promotion, dense re-emit, and hostile-stream validation; adds insert round-trip cases.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
ClickHouse.Driver.Tcp/Types/LowCardinalityColumn.cs	Adds dense in-memory column for `LowCardinality(T)` (dictionary + per-row keys) and supports zero-copy re-emission on write.
ClickHouse.Driver.Tcp/Types/ColumnCodecRegistry.cs	Registers the `LowCardinality` codec factory in the default TCP codec registry.
ClickHouse.Driver.Tcp/Types/Codecs/LowCardinalityShapes.cs	Adds per-element-type shape caching for low-cardinality bridging.
ClickHouse.Driver.Tcp/Types/Codecs/LowCardinalityShape.cs	Implements the typed shape bridge (wrap/read/write/measure) and special byte[] dedup comparer for `FixedString`.
ClickHouse.Driver.Tcp/Types/Codecs/LowCardinalityColumnCodec.cs	Implements the non-generic low-cardinality codec (state prefix, metadata/dictionary/keys body, validation).
ClickHouse.Driver.Tcp/Types/Codecs/ILowCardinalityShape.cs	Defines the shape bridge interface used by the codec to handle generic operations.
ClickHouse.Driver.Tcp/Format/BlockReader.cs	Updates documentation comment clarifying state-prefix behavior for zero-row blocks with dictionary-bearing types.
ClickHouse.Driver.Tcp.Tests/Utilities/InsertRoundTripCase.cs	Adds insert round-trip coverage for `LowCardinality(String/UInt32/FixedString)` and `Array(LowCardinality(String))`.
ClickHouse.Driver.Tcp.Tests/Types/NullableColumnCodecTests.cs	Adjusts an unsupported-inner test case now that `LowCardinality` is supported.
ClickHouse.Driver.Tcp.Tests/Types/LowCardinalityColumnCodecTests.cs	Adds focused unit tests for encoding/decoding, promotion rules, dense re-emit, and hostile-stream validation.
ClickHouse.Driver.Tcp.Tests/Types/ColumnCodecRegistryTests.cs	Updates “unsupported but well-formed type” coverage away from `LowCardinality` now that it is supported.

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

+    }
+
+    /// <inheritdoc/>
+    public T this[int row] => cache is not null ? cache[row] : dictionary[keys[row]];


+    }
+
+    /// <inheritdoc/>
+    public T? this[int row] => cache is not null ? cache[row] : keys[row] == 0 ? null : dictionary[keys[row]];


+    }
+
+    /// <inheritdoc/>
+    public T this[int row] => cache is not null ? cache[row] : keys[row] == 0 ? null : dictionary[keys[row]];


+        // A variable-width inner must measure the value; wrap the single row so the inner codec can price it.
+        T value = ((IColumn<T>)column)[row];
+        var wrapped = ArrayColumn<T>.OverBuffer(column.Name, inner.TypeName, new[] { value }, 1);
+        return maxKeyBytes + inner.MeasureRowBytes(wrapped, 0);


+        // A variable-width inner must measure the value; a NULL row measures the inner default it reserves.
+        T value = IsNull(column, row) ? (T)inner.NullPlaceholderAs(typeof(T)) : Value(column, row);
+        var wrapped = ArrayColumn<T>.OverBuffer(column.Name, inner.TypeName, new[] { value }, 1);
+        return maxKeyBytes + inner.MeasureRowBytes(wrapped, 0);


codecov · 2026-07-28T19:04:45Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

… TCP client Implements LowCardinality for the TCP/Native client: a block-local dictionary of distinct values plus per-row keys indexing it, for both non-nullable and nullable inner types. The serialization-state prefix is a fixed version marker (Int64 = 1); the metadata word, dictionary, and keys live in the column body, so no codec contract change was needed — top-level and under a composite both work with the existing per-column prefix->body sequence. Key width is the smallest unsigned integer that indexes the dictionary (UInt8/16/32); its selection mirrors the reference client's strict-less-than thresholds. Surfaced by a dense LowCardinalityColumn<T> (dictionary column + keys, the zero-copy write source); the ergonomic write source is a plain IColumn<T>, deduplicated into a fresh dictionary on write (structural equality for byte[]/FixedString). The non-nullable dictionary reserves dict[0] for the inner default. LowCardinality(Nullable(T)): the dictionary is still serialized as the bare inner type (no null-map stream); nullability is expressed positionally by two reserved leading slots — dict[0] the NULL marker, dict[1] the inner default — with every NULL row's key pointing at dict[0]. The codec resolves the *bare* inner codec (not the Nullable codec, which would frame a null-map inside the dictionary stream and desynchronize the reader), carries a `nullable` flag, and surfaces the inner element type made nullable: T? for a value inner (NullableLowCardinalityValueColumn) and the nullable reference for a reference inner (NullableLowCardinalityReferenceColumn), reading a `key == 0` row back as NULL. On write, a present value equal to the inner default reuses the reserved default slot 1 rather than adding a duplicate, keeping a present default distinct from NULL. The dense nullable columns are the zero-copy re-emit source via a new IDenseLowCardinality<T> marker, which the non-nullable LowCardinalityColumn<T> deliberately does not implement (its single reserved slot would be misread as NULL under a nullable codec). The server rejects Nullable(LowCardinality(T)), so — like Array/Tuple/Map/Nested — nullability composes inside as LowCardinality(Nullable(T)). Works under composites (Array(LowCardinality(T)) verified). Hostile-stream validation rejects unknown key-width codes, the global-dictionary bit, a missing additional-keys bit, dict_size/key-stream overflow, keys_count mismatch, and out-of-range keys. Verified end-to-end against ClickHouse 26.6: LowCardinality(T) and LowCardinality(Nullable(T)) round-trips for String, UInt32, and FixedString(4), interleaving NULLs with present values (including a present inner-default value). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

LowCardinality builds its dictionary and index stream directly from the ergonomic (optionally nullable) source, with no densify pre-pass or byte measurement. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PR 456 feedback: - ByteArrayEqualityComparer.GetHashCode now hashes a null key to zero deterministically instead of faulting in HashCode.AddBytes. - ILowCardinalityShape.WriteBody doc now covers both dense re-emit sources: the non-nullable LowCardinalityColumn<T> and the nullable columns implementing IDenseLowCardinality<T>. - Correct the LowCardinality(Nullable(T)) comment in InsertRoundTripCase, which called the feature not-yet-supported though cases exercise it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Three codec round-trips asserted values the LowCardinality cases in InsertRoundTripCase already prove through a real INSERT/SELECT: ..._FixedWidthInnerRoundTrips -> "LowCardinality(UInt32)" (superset) ..._NullableStringRoundTrips -> "LowCardinality(Nullable(String))" (identical array) ..._NullableValueTypeRoundTrips -> "LowCardinality(Nullable(UInt32))" (same rows) Everything else in this file stays: the wire-byte tests pinning the state prefix, dictionary layout, key-width selection and the reserved NULL slot are exactly the things a server round-trip cannot observe, since the server re-encodes the dictionary it receives. ReadColumn_WriteThenRead_StringRoundTrips became Values_MaterializesTheDictionaryBackedCacheAndAgreesWithGetValue. GetValue routes to the indexer, which short-circuits on `cache is not null ? cache[row] : dictionary[keys[row]]`, and AssertColumnsEqual only calls GetValue -- so the lazily materialized ArrayPool-rented Values cache ran nowhere. It now reads Values twice (cache reuse) and then GetValue (warm branch), which no test compared before. Two integration cases added: - 300 distinct values, forcing the client to widen the key stream from UInt8 to UInt16. Unit tests proved the client picks that width; with no case above three distinct values, nothing proved the server accepts a client-written wide key stream. It does. - Array(LowCardinality(Nullable(String))), putting the reserved key-0-is-NULL dictionary underneath array offsets, where a reserved-slot off-by-one would surface. Neither half-case reached it. Verified against ClickHouse 26.6 -- 918 tests pass, integration included. Co-Authored-By: Claude <noreply@anthropic.com>

All three low-cardinality columns read keys[row] straight out of the keys buffer in their indexer's uncached branch. That buffer is normally a pooled array longer than the column, and a stale key left in its tail by a previous read is a perfectly valid dictionary index — so a row past RowCount did not fail, it returned a real value from the dictionary. Indexing through the RowCount-sliced Keys span makes that a bounds failure. The Values loops were already bounded by rowCount, so only the indexer changes. The row count stays a parameter: the dictionary holds one entry per distinct value (plus the reserved NULL and default slots), so it says nothing about the column's height. The constructors validate the keys length instead. Co-Authored-By: Claude <noreply@anthropic.com>

Add ILowCardinalityColumn<T>, the public read surface for the dictionary and per-row keys underneath a LowCardinality(T) or LowCardinality(Nullable(T)). This is the type with the widest gap between the two surfaces: the materialized one resolves every row to its dictionary entry, so N rows over a K-entry dictionary produce N values, discarding the very structure the encoding exists for. IDenseLowCardinality<T> now derives from it and carries no members of its own. It never was a read surface — it is the marker for "this column's dictionary may be re-emitted verbatim under a *nullable* codec", which is why the non-nullable column deliberately does not implement it: its dictionary reserves one leading slot where a nullable dictionary reserves two. Splitting the read surface out states that distinction instead of leaving it implied by which accessors happen to exist. Cover the exclusion with an integration test, since violating it corrupts data silently. Only a row whose key is 0 can detect it — slot 0 is an ordinary default value to a non-nullable dictionary and the NULL marker to a nullable one, so rows keyed above the reserve round-trip fine either way and a test without a slot-0 row would pass whether or not the bug is present.

Whether a key of 0 means NULL or an ordinary default depends on the inner type's nullability, and the interface previously told callers to settle that by checking IColumn.TypeName or comparing against the materialized value. That is a poor contract for something the driver already knows statically — the decoder picks a different column class per shape — and getting it wrong is the same silent corruption the write-side exclusion guards against, just on the read side. Expose ReservedSlotCount instead: 1 for a non-nullable inner, 2 for a nullable one. Real distinct values begin at that index, and a row is NULL exactly when it is 2 and the key is 0. Added now rather than later, since widening a shipped interface breaks implementers. Also drop the unqualified "zero-copy" and note the dictionary column is the block's to dispose, matching the other read surfaces.

The map's value composites were Nullable, Array and Tuple-of-leaves, none of which emit a state prefix of their own, so nothing proved the map emits the value stream's prefix. A LowCardinality value does, and its dictionary has to span the whole value run rather than restart per row. Co-Authored-By: Claude <noreply@anthropic.com>

github-actions Bot added triage:feature risk:high labels Jul 22, 2026

alex-clickhouse changed the title ~~Add LowCardinality(T) support for the TCP client~~ TCP J1: Add LowCardinality(T) support Jul 22, 2026

alex-clickhouse requested a review from Copilot July 22, 2026 08:18

Copilot started reviewing on behalf of alex-clickhouse July 22, 2026 08:18 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread ClickHouse.Driver.Tcp/Types/Codecs/LowCardinalityShape.cs

Comment thread ClickHouse.Driver.Tcp/Types/Codecs/LowCardinalityColumnCodec.cs

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from 5e77e71 to a48bd7e Compare July 22, 2026 10:47

alex-clickhouse changed the title ~~TCP J1: Add LowCardinality(T) support~~ TCP J1: Add LowCardinality(T) and LowCardinality(Nullable(T)) support Jul 22, 2026

alex-clickhouse requested a review from Copilot July 22, 2026 10:48

Copilot started reviewing on behalf of alex-clickhouse July 22, 2026 10:49 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from a48bd7e to 8eef24c Compare July 22, 2026 13:23

alex-clickhouse mentioned this pull request Jul 22, 2026

TCP J3: Add Variant(...) support #459

Draft

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from 8eef24c to 56ec266 Compare July 22, 2026 15:26

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch 2 times, most recently from 167437b to 78f63e9 Compare July 23, 2026 08:47

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from 78f63e9 to 8ed03f0 Compare July 23, 2026 09:01

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from 8ed03f0 to 50b13cf Compare July 23, 2026 11:47

alex-clickhouse changed the title ~~TCP J1: Add LowCardinality(T) and LowCardinality(Nullable(T)) support~~ TCP J1/J2: Add LowCardinality(T) and LowCardinality(Nullable(T)) support Jul 23, 2026

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from 50b13cf to 496b0a7 Compare July 23, 2026 18:04

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from 496b0a7 to e0684c8 Compare July 24, 2026 07:19

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from d871af4 to 60eeff4 Compare July 28, 2026 16:01

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from 60eeff4 to 1e500f1 Compare July 28, 2026 18:59

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from 1e500f1 to a3c637d Compare July 28, 2026 19:19

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from a3c637d to bdccf3d Compare July 29, 2026 07:05

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from bdccf3d to 0e73820 Compare July 29, 2026 13:26

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from 0e73820 to 9caf366 Compare July 29, 2026 15:47

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from 9caf366 to 015852a Compare July 30, 2026 08:37

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from 015852a to 4cd27a4 Compare July 30, 2026 09:10

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch 2 times, most recently from f0c2902 to 49fcda6 Compare July 31, 2026 16:14

alex-clickhouse and others added 8 commits August 4, 2026 17:12

Ergonomic LowCardinality write; drop densify/measure

2726b59

LowCardinality builds its dictionary and index stream directly from the ergonomic (optionally nullable) source, with no densify pre-pass or byte measurement. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

alex-clickhouse force-pushed the tcp/epic-j1-lowcardinality branch from 49fcda6 to 46e23bb Compare August 4, 2026 15:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

TCP J1/J2: Add LowCardinality(T) and LowCardinality(Nullable(T)) support - #456

TCP J1/J2: Add LowCardinality(T) and LowCardinality(Nullable(T)) support#456
alex-clickhouse wants to merge 8 commits into
tcp/epic-i5-nestedfrom
tcp/epic-j1-lowcardinality

alex-clickhouse commented Jul 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

codecov Bot commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

alex-clickhouse commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Design

LowCardinality(Nullable(T))

Testing

Uh oh!

github-actions Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Triage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

codecov Bot commented Jul 28, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

alex-clickhouse commented Jul 22, 2026 •

edited

Loading

`LowCardinality(Nullable(T))`

github-actions Bot commented Jul 22, 2026 •

edited

Loading