feat(spanner): prime scaled-up dynamic channel pool channels with SELECT 1 - #14254
Conversation
…ECT 1 The grpc-gcp dynamic channel pool (DCP) now supports a GcpChannelPrimer that runs before a scaled-up channel is published to the pool, but java-spanner never registered one, so channels added under load were published cold: the first requests on them paid for the TCP handshake, TLS session, and server-side connection setup at the moment the pool was already saturated. The Go Spanner client primes scaled-up channels by executing SELECT 1 with the multiplexed session through the new channel. This change brings java-spanner to parity.
There was a problem hiding this comment.
Code Review
This pull request implements channel priming for the dynamic channel pool in the Java Spanner client. It introduces the DynamicChannelPoolPrimer class to execute SELECT 1 on newly scaled-up channels using multiplexed sessions, ensuring connections are established before serving live traffic. It also adds owner ticket registration to safely associate multiplexed sessions with active database clients, and updates SpannerOptions and GapicSpannerRpc to support and configure this priming behavior. Feedback on the pull request points out missing imports for CallCredentials and MoreCallCredentials in GapicSpannerRpc.java, which would cause compilation failures.
| import com.google.auth.Credentials; | ||
| import com.google.cloud.RetryHelper; | ||
| import com.google.cloud.RetryHelper.RetryHelperException; | ||
| import com.google.cloud.grpc.GcpChannelPrimer; |
There was a problem hiding this comment.
The classes io.grpc.CallCredentials and io.grpc.auth.MoreCallCredentials are used in createChannelPrimer but are not imported in this file, which will cause a compilation error. Please add the missing imports.
| import com.google.cloud.grpc.GcpChannelPrimer; | |
| import com.google.cloud.grpc.GcpChannelPrimer; | |
| import io.grpc.CallCredentials; | |
| import io.grpc.auth.MoreCallCredentials; |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements a dynamic channel pool priming mechanism for the Google Cloud Spanner Java client. When dynamic channel pooling is enabled, newly scaled-up channels are primed by executing a SELECT 1 query using a multiplexed session before they are published to the pool, ensuring transport, TLS, and server-side connections are established. The changes introduce the DynamicChannelPoolPrimer class, integrate it with MultiplexedSessionDatabaseClient and GapicSpannerRpc, and add corresponding configuration options and tests. The review feedback highlights two important improvements: first, lazily retrieving credentials inside the CallCredentialsProvider lambda to prevent blocking the startup thread or failing client creation eagerly; second, wrapping the priming execution in a try-catch block to ensure any synchronous exceptions are safely propagated as failed futures.
| static CallCredentialsProvider createChannelPrimeCallCredentialsProvider( | ||
| CredentialsProvider credentialsProvider, | ||
| @Nullable CallCredentialsProvider callCredentialsProvider) { | ||
| final CallCredentials defaultCallCredentials; | ||
| try { | ||
| Credentials credentials = credentialsProvider.getCredentials(); | ||
| defaultCallCredentials = credentials == null ? null : MoreCallCredentials.from(credentials); | ||
| } catch (IOException e) { | ||
| throw newSpannerException(e); | ||
| } | ||
| if (callCredentialsProvider == null) { | ||
| return defaultCallCredentials == null ? null : () -> defaultCallCredentials; | ||
| } | ||
| return () -> { | ||
| CallCredentials callCredentials = callCredentialsProvider.getCallCredentials(); | ||
| return callCredentials != null ? callCredentials : defaultCallCredentials; | ||
| }; | ||
| } |
There was a problem hiding this comment.
Eagerly calling credentialsProvider.getCredentials() during RPC initialization can block the startup thread or fail client creation if there is a transient issue or if the credentials provider is not yet ready. It also prevents dynamic/custom credentials providers from returning fresh credentials if they are swapped or updated, as the first instance is cached forever.
Making the call to credentialsProvider.getCredentials() lazy inside the returned CallCredentialsProvider lambda resolves these issues and improves robustness.
| static CallCredentialsProvider createChannelPrimeCallCredentialsProvider( | |
| CredentialsProvider credentialsProvider, | |
| @Nullable CallCredentialsProvider callCredentialsProvider) { | |
| final CallCredentials defaultCallCredentials; | |
| try { | |
| Credentials credentials = credentialsProvider.getCredentials(); | |
| defaultCallCredentials = credentials == null ? null : MoreCallCredentials.from(credentials); | |
| } catch (IOException e) { | |
| throw newSpannerException(e); | |
| } | |
| if (callCredentialsProvider == null) { | |
| return defaultCallCredentials == null ? null : () -> defaultCallCredentials; | |
| } | |
| return () -> { | |
| CallCredentials callCredentials = callCredentialsProvider.getCallCredentials(); | |
| return callCredentials != null ? callCredentials : defaultCallCredentials; | |
| }; | |
| } | |
| static CallCredentialsProvider createChannelPrimeCallCredentialsProvider( | |
| CredentialsProvider credentialsProvider, | |
| @Nullable CallCredentialsProvider callCredentialsProvider) { | |
| return () -> { | |
| if (callCredentialsProvider != null) { | |
| CallCredentials callCredentials = callCredentialsProvider.getCallCredentials(); | |
| if (callCredentials != null) { | |
| return callCredentials; | |
| } | |
| } | |
| try { | |
| Credentials credentials = credentialsProvider.getCredentials(); | |
| return credentials == null ? null : MoreCallCredentials.from(credentials); | |
| } catch (IOException e) { | |
| throw newSpannerException(e); | |
| } | |
| }; | |
| } |
References
- Prefer lazy initialization over eager initialization for resource-intensive or blocking operations if they are not guaranteed to be used immediately, to avoid unnecessary performance overhead.
There was a problem hiding this comment.
constructor already resolves credentials eagerly through GAX
| @Override | ||
| public ListenableFuture<Void> prime(ManagedChannel channel) { | ||
| String sessionName = getPrimeSessionName(); | ||
| if (sessionName == null) { | ||
| // The primer cannot gate the pool's scale-up decision, so the attempt fails fast. The pool's | ||
| // retry with backoff and its close-on-failure behaviour handle the unavailable session. | ||
| return Futures.immediateFailedFuture( | ||
| SpannerExceptionFactory.newSpannerException( | ||
| ErrorCode.FAILED_PRECONDITION, | ||
| "Cannot prime a dynamic channel pool channel before a multiplexed session is" | ||
| + " available")); | ||
| } | ||
| return executePrimeStatement(channel, sessionName); | ||
| } |
There was a problem hiding this comment.
The prime method returns a ListenableFuture<Void>, but executePrimeStatement can throw synchronous exceptions (e.g., if callCredentialsProvider.getCallCredentials() throws a SpannerException or if metadata/headers generation fails). Synchronous exceptions from future-returning methods can bypass standard future error handling and crash calling threads/executors.
Wrapping the call to executePrimeStatement in a try-catch block and returning Futures.immediateFailedFuture(t) ensures that all failures are propagated safely as failed futures.
@Override
public ListenableFuture<Void> prime(ManagedChannel channel) {
String sessionName = getPrimeSessionName();
if (sessionName == null) {
// The primer cannot gate the pool's scale-up decision, so the attempt fails fast. The pool's
// retry with backoff and its close-on-failure behaviour handle the unavailable session.
return Futures.immediateFailedFuture(
SpannerExceptionFactory.newSpannerException(
ErrorCode.FAILED_PRECONDITION,
"Cannot prime a dynamic channel pool channel before a multiplexed session is"
+ " available"));
}
try {
return executePrimeStatement(channel, sessionName);
} catch (Throwable t) {
return Futures.immediateFailedFuture(t);
}
}References
- Ensure that any future-returning method guarantees completion (either successfully or exceptionally) even if synchronous exceptions are thrown during initialization. Wrap the execution in a try-catch block to handle failures and return a failed future.
There was a problem hiding this comment.
It already catches Throwable
Ensure the primer API boundary converts synchronous failures into failed futures. Restore the built-in metrics test to the merged upstream emulator-host setup.
🤖 I have created a release *beep* *boop* --- <details><summary>1.91.0</summary> ## [1.91.0](v1.90.0...v1.91.0) (2026-09-04) ### ⚠ BREAKING CHANGES * **datalabeling:** remove java-datalabeling library ([#14189](#14189)) * **datacatalog:** remove java-datacatalog library ([#14178](#14178)) ### Features * **apptopology:** onboard a new library ([#14204](#14204)) ([f871a07](f871a07)) * **bigtable:** Option to disable direct access fallback ([#14193](#14193)) ([5b6a974](5b6a974)) * **datacatalog:** remove java-datacatalog library ([#14178](#14178)) ([2910f63](2910f63)) * **datalabeling:** remove java-datalabeling library ([#14189](#14189)) ([334d35a](334d35a)), refs [#14176](#14176) * **gax:** add getSingleHeader to HttpHeadersUtils ([#14137](#14137)) ([f114422](f114422)) * **gax:** add ResumableUploadClient.startUpload() and supporting types ([#14138](#14138)) ([d7e2be2](d7e2be2)) * **gax:** add ResumableUploadResponseParser ([#14135](#14135)) ([e1a0b17](e1a0b17)) * **gax:** allow non-JSON HttpContent and absolute request URLs in HttpRequestRunnable ([#14134](#14134)) ([ac4a49a](ac4a49a)) * **gax:** implement queryStatusCallable for resumable uploads ([#14155](#14155)) ([67719c0](67719c0)) * **gax:** implement startUploadCallable for resumable uploads ([#14139](#14139)) ([4e65b6c](4e65b6c)) * **gax:** implement uploadChunkCallable for resumable uploads ([#14140](#14140)) ([1c643de](1c643de)) * **google/shopping/merchant/loyaltycustomers/v1:** onboard new library ([#14257](#14257)) ([590a51c](590a51c)) * **grpc-gcp:** drain scaled-down channels ([#14216](#14216)) ([bbbd18c](bbbd18c)) * **grpc-gcp:** move scale-up to background worker ([#14206](#14206)) ([41f0a2e](41f0a2e)) * **grpc-gcp:** penalize retryable channel errors ([#14219](#14219)) ([829872e](829872e)) * **grpc-gcp:** prime scaled channels before publish ([#14232](#14232)) ([dd75645](dd75645)) * **spanner:** add client-level CallContextConfigurator to SpannerOptions ([#14256](#14256)) ([eef74d2](eef74d2)) * **spanner:** prime scaled-up dynamic channel pool channels with SELECT 1 ([#14254](#14254)) ([4d26645](4d26645)) ### Bug Fixes * **auth:** refine JSpecify nullability annotations for external account credentials ([#14164](#14164)) ([385e1f2](385e1f2)) * **bigquery-jdbc:** resolve `statementType` via `getJob` fallback to avoid post-execution dry run for DDL ([#14265](#14265)) ([d38d9f0](d38d9f0)) * **bigtable:** truncate client generated timestamps on the emulator c… ([#14234](#14234)) ([94e680f](94e680f)) * **gax:** propagate structured LRO error details to ApiException ([#14022](#14022)) ([865a15b](865a15b)) * **grpc-gcp:** correct channel lifecycle bookkeeping ([#14196](#14196)) ([3df32ba](3df32ba)) * **grpc-gcp:** prevent hot-channel skew ([#14198](#14198)) ([07a7505](07a7505)) * **java-sql:** temp fix for lint and skip generate ([#14005](#14005)) ([b6f73db](b6f73db)) * **spanner:** prioritize leader replica for read-write transactions in location-aware routing ([#14195](#14195)) ([6383f81](6383f81)) * **spanner:** route ExecuteBatchDml through transaction affinity endpoint ([#14192](#14192)) ([5e9e3cd](5e9e3cd)) * **storage:** resolve GraalVM Native Image test failure for java-storage ([#14226](#14226)) ([93e0920](93e0920)) ### Performance Improvements * **bigquery-jdbc:** eliminate sync getJob RPC call ([#14197](#14197)) ([ccf12b0](ccf12b0)) * **ci:** replace mvn help:evaluate with native bash and sed extraction ([#14218](#14218)) ([744a522](744a522)) ### Dependencies * Upgrade grpc-java to 1.83.0 ([#13967](#13967)) ([0cdc695](0cdc695)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
The grpc-gcp dynamic channel pool (DCP) now supports a GcpChannelPrimer that runs before a scaled-up channel is published to the pool, but java-spanner never registered one, so channels added under load were published cold: the first requests on them paid for the TCP handshake, TLS session, and server-side connection setup at the moment the pool was already saturated. The Go Spanner client primes scaled-up channels by executing SELECT 1 with the multiplexed session through the new channel. This change brings java-spanner to parity.