TCP N: Add ClickHouseTcpClient: public client API (read path + inserts) by alex-clickhouse · Pull Request #462 · ClickHouse/clickhouse-cs

alex-clickhouse · 2026-07-24T11:42:44Z

First branch of Epic N (client API) for the native-TCP client. Adds the user-facing entry point on top of the existing raw connection, plus the options and connection-acquisition plumbing. Stacked on tcp/epic-j4-dynamic.

What's here

ClickHouseTcpClient — [Experimental("CHTCP0001")], IAsyncDisposable, safe to share:
- StreamAsync → IAsyncEnumerable<Block> (low-level columnar tier)
- QueryAsync → IAsyncEnumerable<object[]> (untyped rows, boxed via IColumn.GetValue)
- ExecuteAsync (non-result statements), InsertAsync (columnar IReadOnlyList<IColumn>), PingAsync
- Auto-enables output_format_native_use_flattened_dynamic_and_json_serialization so Dynamic/JSON decode without the caller knowing (caller value still wins).
Options — ClickHouseTcpClientOptions + ClickHouseTcpConnectionStringBuilder (Host/Port/Username/Password/Database/QuotaKey/DialTimeout/ReadTimeout/MaxSendBufferBytes + set_<name> custom settings), and a minimal ClickHouseTcpQueryOptions (QueryId + Settings).
Connection seam — internal IConnectionSource/IConnectionLease with an interim SingleConnectionSource (one connection, serialized, redials a terminated one). DialTimeout bounds connect+handshake. A real pool (Epic M) implements the same interface with no client change.
MaxSendBufferBytes threaded through InsertAsync as the between-column flush threshold (write memory backstop), independent of the 50 MB block-split target.
Block is now public (constructor + Info stayed internal so BlockInfo isn't leaked); added Block.ColumnNames.

Streaming release semantics

StreamAsync rents a connection and returns it to the source exactly once on full drain, early enumerator disposal, or exception (an Interlocked guard prevents double-return; a terminated connection is discarded and redialed on the next rent).

Tests

Unit: options validation + handshake mapping, connection-string parsing (set_* custom settings, defaults, round-trip), settings-merge (N1a injection / caller-wins), SingleConnectionSource lifecycle (idempotent dispose, rent-after-dispose, pre-cancelled token).
Integration (live server): streaming, early-dispose→redial reuse, server-error→still-usable, object[] rows + owned-row retention, ExecuteAsync round-trip, columnar insert round-trip, schema-mismatch, per-query settings, Dynamic decode without the caller setting the flag (proves N1a), tiny 4 KB send-buffer flushing 20k rows intact (proves MaxSendBufferBytes), concurrency, connection-string construction.

Full net9.0 suite green (1101 tests). Coverage ~93% line / ~87% branch on the new code.

Deferred (called out)

Row-oriented / POCO insert (N9) + POCO read (N5/N5a) → Branch 2.
Query-parameter binding & value formatting (N11) + rich per-query options (N10) → Branch 3.
ReadTimeout is parsed/stored but not yet enforced (it is the idle read-loop deadline of Q3).
The client can throw the still-internal ClickHouseServerException/ClickHouseProtocolException (callers see the base Exception) — exception hierarchy is Q1/Epic R.

🤖 Generated with Claude Code

github-actions · 2026-07-24T11:44:20Z

Triage

Category: feature • Risk: high

Summary
This PR (first branch of "Epic N") introduces ClickHouseTcpClient, a new user-facing high-level client over the native TCP protocol. It adds ClickHouseTcpClientOptions, ClickHouseTcpConnectionStringBuilder, ClickHouseTcpQueryOptions, an internal IConnectionSource/IConnectionLease seam (with a SingleConnectionSource interim implementation), and partially publicizes Block (Block.ColumnNames). The client exposes StreamAsync (columnar IAsyncEnumerable<Block>), QueryAsync (untyped object[] rows), ExecuteAsync, InsertAsync, and PingAsync. All public types are marked [Experimental("CHTCP0001")]. Diff is +1637/-5 lines, almost entirely net-new code in ClickHouse.Driver.Tcp/ and its test project.

What this impacts

New public API surface in ClickHouse.Driver.Tcp: ClickHouseTcpClient, ClickHouseTcpClientOptions, ClickHouseTcpConnectionStringBuilder, ClickHouseTcpQueryOptions, Block.ColumnNames (all [Experimental])
Internal IConnectionSource/IConnectionLease/SingleConnectionSource — the connection-lifecycle seam for the future pool
Test infrastructure: TcpServerFixture extended with Options(), CreateClient(), ConnectionString helpers

Concerns

Concurrency rule fires: SingleConnectionSource uses an Interlocked guard against double-return of a connection lease, and a serializing gate (SemaphoreSlim or equivalent) for concurrent RentAsync calls. The test RentAsync_PreCancelledToken_ThrowsOperationCanceledAndDoesNotDeadlock names a deadlock scenario explicitly — this is exactly the pattern the concurrency rubric rule targets.
ReadTimeout stored but not enforced: callers who set it may silently get no timeout behavior; could surprise users and should be documented at the call site or validated differently if enforcement is genuinely deferred.
PR is in DRAFT: not yet ready for final review; triage provided as early signal only.
Block partially publicized: Block.ColumnNames added to public surface. Verify PublicAPI/*.txt was updated (not visible in diff summary).

Required reviewer action

High risk: PR body must include an architectural description before review. The existing body is thorough (streaming release semantics, connection-seam design, deferred items); reviewer should confirm it covers the concurrency model of SingleConnectionSource (gate acquisition/release under cancellation) before approving.

Copilot

Pull request overview

Adds an experimental, high-level native-TCP client API (ClickHouseTcpClient) on top of the existing TCP protocol layer, including connection acquisition plumbing and client/query options, with tests covering core behaviors and live-server integration.

Changes:

Introduces ClickHouseTcpClient with streaming (Block), untyped row streaming (object[]), execute/insert, ping, and per-query settings merge (incl. flattened Dynamic/JSON serialization injection).
Adds options and parsing/building support (ClickHouseTcpClientOptions, ClickHouseTcpQueryOptions, ClickHouseTcpConnectionStringBuilder) plus a connection-source seam with an initial SingleConnectionSource.
Makes Block public and adds Block.ColumnNames; threads MaxSendBufferBytes through the TCP insert write path; adds unit + integration tests.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
ClickHouse.Driver.Tcp/Protocol/ClickHouseTcpConnection.cs	Threads a send-buffer flush threshold through insert streaming and validates it.
ClickHouse.Driver.Tcp/Format/Block.cs	Makes `Block` public and adds cached `ColumnNames`.
ClickHouse.Driver.Tcp/Client/SingleConnectionSource.cs	Implements a serialized, single-connection rent/lease source with redial on terminated connections.
ClickHouse.Driver.Tcp/Client/IConnectionSource.cs	Defines internal connection source + lease interfaces for pooling seam.
ClickHouse.Driver.Tcp/Client/ClickHouseTcpQueryOptions.cs	Adds minimal per-query overrides (QueryId + settings).
ClickHouse.Driver.Tcp/Client/ClickHouseTcpConnectionStringBuilder.cs	Adds TCP-native connection string builder/parser (incl. `set_*` settings).
ClickHouse.Driver.Tcp/Client/ClickHouseTcpClientOptions.cs	Defines validated client-level endpoint/timeout/buffer/settings options.
ClickHouse.Driver.Tcp/Client/ClickHouseTcpClient.cs	Adds the experimental public TCP client API and settings merge behavior.
ClickHouse.Driver.Tcp.Tests/Integration/TcpServerFixture.cs	Adds fixture helpers for creating options/client and a TCP connection string.
ClickHouse.Driver.Tcp.Tests/Integration/ClickHouseTcpClientIntegrationTests.cs	Live-server integration coverage for streaming, inserts, reuse/redial, settings, and concurrency.
ClickHouse.Driver.Tcp.Tests/Client/SingleConnectionSourceTests.cs	Unit tests for `SingleConnectionSource` lifecycle and cancellation behavior.
ClickHouse.Driver.Tcp.Tests/Client/ClickHouseTcpConnectionStringBuilderTests.cs	Unit tests for builder parsing/defaults/custom settings/round-trip.
ClickHouse.Driver.Tcp.Tests/Client/ClickHouseTcpClientSettingsTests.cs	Unit tests for settings merge + flattened serialization injection behavior.
ClickHouse.Driver.Tcp.Tests/Client/ClickHouseTcpClientOptionsTests.cs	Unit tests for defaults, validation, and handshake mapping.
ClickHouse.Driver.Tcp.Tests/ClickHouse.Driver.Tcp.Tests.csproj	Suppresses the experimental API diagnostic for tests.

alex-clickhouse · 2026-07-24T12:04:35Z

Thanks — addressed all three (pushed as an amend to 1fa7db1):

Connection-string builder typed getters (real bug) — GetIntOrDefault/GetTimeSpanSecondsOrDefault only matched string, but the typed setters store boxed int/double, so a set-then-get on the same builder returned the default. Getters now handle both the boxed typed value and the parsed string (invariant culture, NumberStyles.Integer/Float).
Block.ColumnNames publication — now builds the array into a local and publishes via Volatile.Write (with Volatile.Read on the getter), so a concurrent reader can't observe the reference before its elements are written. A benign double-compute yields equivalent arrays, so only the torn publication needed guarding.
Test gap — added TypedSetters_ReadBackOnSameInstance_ReturnValuesNotDefaults, which asserts Port/MaxSendBufferBytes/DialTimeout/ReadTimeout read back on the same instance (this is what catches Bump Vampire/setup-wsl from 5 to 6 #1; the prior tests only round-tripped through ConnectionString, which stringifies and masked it).

Full net9.0 suite green (1102 tests).

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

ClickHouse.Driver.Tcp/Client/ClickHouseTcpClient.cs:234

Settings dictionaries can currently contain an empty key or a null value. In the native protocol, settings are encoded as (key, flags, value) triples terminated by an empty key, so an empty key can corrupt the packet; a null value will also throw when the Query packet is written. Consider validating during merge so failures are deterministic and actionable (covers both client-level and per-query settings).

            foreach (KeyValuePair<string, string> entry in clientSettings)
            {
                merged[entry.Key] = entry.Value;
            }

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

+    private string GetStringOrDefault(string name, string @default)
+        => TryGetValue(name, out object value) && value is string s ? s : @default;


@@ -15,14 +16,16 @@ namespace ClickHouse.Driver.Tcp.Format;
 /// <c>Values.ToArray()</c>) while iterating.


+        var merged = new Dictionary<string, string>(StringComparer.Ordinal);
+        if (clientSettings is not null)
+        {
+            foreach (KeyValuePair<string, string> entry in clientSettings)
+            {
+                merged[entry.Key] = entry.Value;
+            }
+        }
+
+        if (perQuerySettings is not null)
+        {
+            foreach (KeyValuePair<string, string> entry in perQuerySettings)
+            {
+                merged[entry.Key] = entry.Value;
+            }
+        }


codecov · 2026-07-28T19:05:29Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Introduces the first user-facing entry point for the native-TCP client on top of the existing raw connection, plus the options and connection-acquisition plumbing it needs. - ClickHouseTcpClient ([Experimental("CHTCP0001")]): StreamAsync (block tier), QueryAsync (object[] rows), ExecuteAsync (non-result statements), InsertAsync (columnar), PingAsync. Safe to share; auto-enables the flattened Dynamic/JSON serialization on every operation (a caller value still wins). - ClickHouseTcpClientOptions + ClickHouseTcpConnectionStringBuilder (Host/Port/Username/Password/Database/QuotaKey/DialTimeout/ReadTimeout/ MaxSendBufferBytes + set_<name> custom settings) and a minimal ClickHouseTcpQueryOptions (QueryId + Settings). - IConnectionSource/IConnectionLease seam with a single-connection interim source that serializes access and redials a terminated connection; a real pool implements the same interface later. DialTimeout bounds connect+handshake. - MaxSendBufferBytes is threaded through InsertAsync as the between-column flush threshold (the write memory backstop), independent of the block-split target. - Block is now public (its constructor and Info stayed internal); added Block.ColumnNames for header-order name lookup. Covered by unit tests (options/connection-string/settings-merge/source lifecycle) and live-server integration tests (streaming, early-dispose redial, server-error reuse, columnar round-trip, per-query settings, Dynamic decode without the caller setting the flag, tiny send-buffer flush, concurrency). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Asserts that writing a column from its ergonomic form produces bytes identical to writing the dense column read back from that same wire output, across Array/Nullable/Tuple/Map. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PR 462 feedback: - Validate client-level CustomSettings at construction: reject an empty/null setting name (it would collide with the empty key that terminates the wire settings list) and a null value. - ToOptions rejects a bare 'set_' key (empty setting name) and never emits a null value for a value-less set_ key. - Copy CustomSettings into an owned dictionary in the client ctor, so a caller mutating their dictionary cannot fault or partially apply a concurrent settings merge on the shared client. - Correct the Password doc: the native transport is unencrypted, so the password is not TLS-protected by this client. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…on, Block doc PR 462 (round 2): - ToOptions now formats a typed set_ value (e.g. builder["set_max_threads"] = 4) as an invariant string instead of silently dropping it to empty. - MergeSettings validates per-query settings (user-provided, unlike client CustomSettings): an empty name would truncate the wire settings list and a null value cannot be written, so both are rejected. - Document that a Block yielded by a query must not be disposed by the consumer — the reader owns its borrowed, pooled storage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions Bot added triage:feature risk:high labels Jul 24, 2026

alex-clickhouse requested a review from Copilot July 24, 2026 11:46

Copilot started reviewing on behalf of alex-clickhouse July 24, 2026 11:46 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread ClickHouse.Driver.Tcp/Client/ClickHouseTcpConnectionStringBuilder.cs Outdated

Comment thread ClickHouse.Driver.Tcp/Format/Block.cs

Comment thread ClickHouse.Driver.Tcp.Tests/Client/ClickHouseTcpConnectionStringBuilderTests.cs

alex-clickhouse force-pushed the tcp/epic-n1-client branch from ec2c0ca to 1fa7db1 Compare July 24, 2026 12:04

alex-clickhouse force-pushed the tcp/epic-n1-client branch from 1fa7db1 to 293f045 Compare July 24, 2026 18:43

alex-clickhouse requested a review from Copilot July 24, 2026 18:44

Copilot started reviewing on behalf of alex-clickhouse July 24, 2026 18:44 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread ClickHouse.Driver.Tcp/Client/ClickHouseTcpConnectionStringBuilder.cs

Comment thread ClickHouse.Driver.Tcp/Client/ClickHouseTcpClientOptions.cs Outdated

Comment thread ClickHouse.Driver.Tcp/Client/ClickHouseTcpClient.cs

alex-clickhouse force-pushed the tcp/epic-n1-client branch from 293f045 to e27b97c Compare July 24, 2026 21:46

alex-clickhouse requested a review from Copilot July 24, 2026 21:46

Copilot started reviewing on behalf of alex-clickhouse July 24, 2026 21:47 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

alex-clickhouse force-pushed the tcp/epic-n1-client branch from e27b97c to 5e94cd5 Compare July 25, 2026 06:47

alex-clickhouse force-pushed the tcp/epic-n1-client branch from 5e94cd5 to 87403eb Compare July 27, 2026 10:58

alex-clickhouse force-pushed the tcp/epic-n1-client branch 2 times, most recently from f623015 to 9585d2a Compare July 28, 2026 15:26

alex-clickhouse force-pushed the tcp/epic-n1-client branch from 9585d2a to f4b752e Compare July 28, 2026 16:01

alex-clickhouse changed the title ~~Add ClickHouseTcpClient: native-TCP client API (read path + inserts)~~ TCP N: Add ClickHouseTcpClient: public client API (read path + inserts) Jul 28, 2026

alex-clickhouse force-pushed the tcp/epic-n1-client branch from f4b752e to 069d77c Compare July 28, 2026 18:59

alex-clickhouse force-pushed the tcp/epic-n1-client branch 2 times, most recently from 0ec04c9 to cff7c93 Compare July 29, 2026 07:05

alex-clickhouse force-pushed the tcp/epic-n1-client branch from cff7c93 to affd863 Compare July 29, 2026 13:26

alex-clickhouse force-pushed the tcp/epic-n1-client branch from affd863 to d768c40 Compare July 29, 2026 15:47

alex-clickhouse force-pushed the tcp/epic-n1-client branch from d768c40 to 3e03228 Compare July 30, 2026 07:53

alex-clickhouse force-pushed the tcp/epic-n1-client branch from 3e03228 to c7ba344 Compare July 30, 2026 08:13

alex-clickhouse force-pushed the tcp/epic-n1-client branch from c7ba344 to 6dc491d Compare July 30, 2026 08:37

alex-clickhouse force-pushed the tcp/epic-n1-client branch from 6dc491d to 8f25aeb Compare July 30, 2026 09:11

alex-clickhouse force-pushed the tcp/epic-n1-client branch from 8f25aeb to 209b0b5 Compare July 30, 2026 09:47

alex-clickhouse force-pushed the tcp/epic-n1-client branch 2 times, most recently from 724dc65 to 9f8fdff Compare August 3, 2026 08:13

alex-clickhouse and others added 4 commits August 4, 2026 17:12

alex-clickhouse force-pushed the tcp/epic-n1-client branch from 9f8fdff to 2b6fead Compare August 4, 2026 15:16

Navigation Menu

Uh oh!

TCP N: Add ClickHouseTcpClient: public client API (read path + inserts) - #462

TCP N: Add ClickHouseTcpClient: public client API (read path + inserts)#462
alex-clickhouse wants to merge 4 commits into
tcp/epic-j4-dynamicfrom
tcp/epic-n1-client

alex-clickhouse commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-clickhouse commented Jul 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

codecov Bot commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		private string GetStringOrDefault(string name, string @default)
		=> TryGetValue(name, out object value) && value is string s ? s : @default;

		@@ -15,14 +16,16 @@ namespace ClickHouse.Driver.Tcp.Format;
		/// <c>Values.ToArray()</c>) while iterating.

Uh oh!

Conversation

alex-clickhouse commented Jul 24, 2026

What's here

Streaming release semantics

Tests

Deferred (called out)

Uh oh!

github-actions Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Triage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-clickhouse commented Jul 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

codecov Bot commented Jul 28, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 24, 2026 •

edited

Loading