Skip to content

feat(bigquery-jdbc): add opt-out controls #14199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Neenu1995 wants to merge 6 commits into from
Open

feat(bigquery-jdbc): add opt-out controls #14199

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4cca913
feat(telemetry): implement PR 8 opt-out controls, enum converters and…
Neenu1995 Aug 25, 2026
3d5c91e
add statement types
Neenu1995 Aug 27, 2026
4f45388
address comments
Neenu1995 Aug 27, 2026
3de37e2
fix property precedence
Neenu1995 Aug 27, 2026
ffcc65d
add logSource
Neenu1995 Sep 3, 2026
bdffd35
remove aggregation logic from this PR
Neenu1995 Sep 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 89 additions & 2 deletions ...dbc/src/main/java/com/google/cloud/bigquery/jdbc/telemetry/v1/TelemetryConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@
package com.google.cloud.bigquery.jdbc.telemetry.v1;

import java.util.Objects;
import java.util.Properties;

/** Configuration settings for the BigQuery JDBC driver telemetry client. */
final class TelemetryConfiguration {
static final boolean DEFAULT_ENABLED = true;
// TODO: change DEFAULT_LOG_SOURCE value once the value is assigned.
static final int DEFAULT_LOG_SOURCE = -1;
static final int DEFAULT_LOG_SOURCE = 3071;
static final String DEFAULT_ENDPOINT_URL = "https://play.googleapis.com/log";
static final long DEFAULT_UPLOAD_INTERVAL_MS = 300_000L;
static final int DEFAULT_BATCH_SIZE_THRESHOLD = 5000;
Expand Down Expand Up @@ -156,6 +156,93 @@ Builder setDriverEnvironment(DriverEnvironment driverEnvironment) {
return this;
}

Builder resolveProperties(Properties connectionProperties) {
// 1. Connection Properties (lowest precedence)
if (connectionProperties != null) {
String propValue = connectionProperties.getProperty("EnableDiagnosticTelemetry");
if (propValue == null) {

@logachev logachev Sep 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This if assigns exact same value

propValue = connectionProperties.getProperty("enableDiagnosticTelemetry");
}
if (propValue != null) {
if ("0".equals(propValue) || "false".equalsIgnoreCase(propValue)) {

@logachev logachev Sep 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have BigQueryJdbcUrlUtility.convertIntToBoolean(), can we reuse it? (or move it to some utils class to use in both places)

this.enabled = false;
} else if ("1".equals(propValue) || "true".equalsIgnoreCase(propValue)) {
this.enabled = true;
}
}

String uploadIntervalStr = connectionProperties.getProperty("TelemetryUploadInterval");
if (uploadIntervalStr != null) {
try {
this.uploadIntervalMs = Long.parseLong(uploadIntervalStr);
} catch (NumberFormatException ignored) {
}
}

String batchSizeStr = connectionProperties.getProperty("TelemetryBatchSize");
if (batchSizeStr != null) {
try {
this.batchSizeThreshold = Integer.parseInt(batchSizeStr);
} catch (NumberFormatException ignored) {
}
}
}

// 2. Environment Variables (overrides connection properties)
String envEnabled = System.getenv("GOOGLE_CLOUD_TELEMETRY_ENABLED");
if (envEnabled != null) {
if ("0".equals(envEnabled) || "false".equalsIgnoreCase(envEnabled)) {
this.enabled = false;
} else if ("1".equals(envEnabled) || "true".equalsIgnoreCase(envEnabled)) {
this.enabled = true;
}
}

String envInterval = System.getenv("GOOGLE_CLOUD_TELEMETRY_UPLOAD_INTERVAL");
if (envInterval != null) {
try {
this.uploadIntervalMs = Long.parseLong(envInterval);
} catch (NumberFormatException ignored) {
}
}

String envBatch = System.getenv("GOOGLE_CLOUD_TELEMETRY_BATCH_SIZE");
if (envBatch != null) {
try {
this.batchSizeThreshold = Integer.parseInt(envBatch);
} catch (NumberFormatException ignored) {
}
}

// 3. JVM System Properties (highest precedence)
String sysEnabled = System.getProperty("GOOGLE_CLOUD_TELEMETRY_ENABLED");
if (sysEnabled != null) {
if ("0".equals(sysEnabled) || "false".equalsIgnoreCase(sysEnabled)) {
this.enabled = false;
} else if ("1".equals(sysEnabled) || "true".equalsIgnoreCase(sysEnabled)) {
this.enabled = true;
}
}

String sysInterval = System.getProperty("GOOGLE_CLOUD_TELEMETRY_UPLOAD_INTERVAL");

@logachev logachev Sep 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we have 2 types here, int & bool. I'd suggest moving it to helper methods to have smth like this

this.uploadIntervalMs = parseInt(System.getProperty("GOOGLE_CLOUD_TELEMETRY_UPLOAD_INTERVAL"), this.UploadIntervalMs);

(Second param for default value)

if (sysInterval != null) {
try {
this.uploadIntervalMs = Long.parseLong(sysInterval);
} catch (NumberFormatException ignored) {
}
}

String sysBatch = System.getProperty("GOOGLE_CLOUD_TELEMETRY_BATCH_SIZE");
if (sysBatch != null) {
try {
this.batchSizeThreshold = Integer.parseInt(sysBatch);
} catch (NumberFormatException ignored) {
}
}

return this;
}

TelemetryConfiguration build() {
return new TelemetryConfiguration(this);
}
Expand Down
141 changes: 140 additions & 1 deletion ...uery-jdbc/src/main/java/com/google/cloud/bigquery/jdbc/telemetry/v1/TelemetryManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,10 @@

package com.google.cloud.bigquery.jdbc.telemetry.v1;

import com.google.cloud.bigquery.JobStatistics.QueryStatistics;
import com.google.cloud.bigquery.jdbc.BigQueryJdbcCustomLogger;
import com.google.protobuf.Descriptors.EnumValueDescriptor;
import java.util.Properties;
import java.util.logging.Level;
import java.util.logging.Logger;

Expand All @@ -32,6 +35,7 @@ final class TelemetryManager implements AutoCloseable {
new BigQueryJdbcCustomLogger(TelemetryManager.class.getName());

private static volatile TelemetryManager instance;
private static volatile boolean globallyDisabled = false;

private final TelemetryBatcher batcher;

Expand All @@ -44,12 +48,36 @@ private TelemetryManager(TelemetryBatcher batcher) {
* and transport.
*/
static TelemetryManager getInstance() {
return getInstance(null);
}

static TelemetryManager getInstance(Properties properties) {
if (globallyDisabled) {
return null;
}

if (properties != null) {
TelemetryConfiguration configCheck =
TelemetryConfiguration.builder().resolveProperties(properties).build();
if (!configCheck.isEnabled()) {
synchronized (TelemetryManager.class) {
globallyDisabled = true;
closeInstance();
}
return null;
}
}

TelemetryManager localRef = instance;
if (localRef == null) {
synchronized (TelemetryManager.class) {
if (globallyDisabled) {
return null;
}
localRef = instance;
if (localRef == null) {
TelemetryConfiguration config = TelemetryConfiguration.builder().build();
TelemetryConfiguration config =
TelemetryConfiguration.builder().resolveProperties(properties).build();

@logachev logachev Sep 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

properties is always null at this point

ClearcutTransport transport = new ClearcutTransport(config);
TelemetryBatcher batcher = new TelemetryBatcher(config, transport);
localRef = new TelemetryManager(batcher);
Expand Down Expand Up @@ -116,4 +144,115 @@ public void close() {
batcher.close();
}
}

// Package-private test helper to reset the global kill switch between test runs
static synchronized void resetGlobalDisableForTest() {
globallyDisabled = false;
}

static StatementType toStatementType(QueryStatistics.StatementType bqStatementType) {
if (bqStatementType == null) {
return StatementType.STATEMENT_TYPE_UNSPECIFIED;
}

EnumValueDescriptor desc =
StatementType.getDescriptor().findValueByName("STATEMENT_TYPE_" + bqStatementType.name());

return desc != null ? StatementType.valueOf(desc) : StatementType.STATEMENT_TYPE_OTHER;
}
Comment thread
Neenu1995 marked this conversation as resolved.

static AuthenticationType toAuthenticationType(int oauthType) {
switch (oauthType) {
case 0:
return AuthenticationType.AUTHENTICATION_TYPE_SERVICE_ACCOUNT;
case 1:
return AuthenticationType.AUTHENTICATION_TYPE_USER_AUTHENTICATION;
case 2:

@logachev logachev Sep 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is int coming from? It doesn't match OAuthType connection properties

return AuthenticationType.AUTHENTICATION_TYPE_APPLICATION_DEFAULT_CREDENTIALS;
case 3:
return AuthenticationType.AUTHENTICATION_TYPE_EXTERNAL;
case 4:
return AuthenticationType.AUTHENTICATION_TYPE_TOKEN;
default:
return AuthenticationType.AUTHENTICATION_TYPE_CUSTOM;
}
}

static final double[] HISTOGRAM_BOUNDS = {
10.0, 50.0, 100.0, 250.0, 500.0, 1000.0, 5000.0, 10000.0
};

static DurationHistogram toDurationBucketMs(long durationMs) {
DurationHistogram.Builder builder =
DurationHistogram.newBuilder().setCount(1).setSum(durationMs);

int bucketIndex = HISTOGRAM_BOUNDS.length;
for (int i = 0; i < HISTOGRAM_BOUNDS.length; i++) {
builder.addExplicitBounds(HISTOGRAM_BOUNDS[i]);
if (bucketIndex == HISTOGRAM_BOUNDS.length && durationMs < HISTOGRAM_BOUNDS[i]) {
bucketIndex = i;
}
}
for (int i = 0; i <= HISTOGRAM_BOUNDS.length; i++) {
builder.addBucketCounts(i == bucketIndex ? 1L : 0L);
}
return builder.build();
}

static void recordConnectionAttempt(Status status, int errorCode, AuthenticationType authType) {
runSafely(
() -> {
TelemetryManager mgr = instance;
if (mgr != null && mgr.getBatcher() != null) {
mgr.getBatcher()
.offerConnectionAttempt(
ConnectionAttempt.newBuilder()
.setStatus(status)
.setErrorCode(errorCode)
.setAuthType(authType)
.setCount(1)
.build());
}
});
}

static void recordStatementExecution(
StatementType statementType,
QueryApiType apiType,
Status status,
int errorCode,
long durationMs) {
runSafely(
() -> {
TelemetryManager mgr = instance;
if (mgr != null && mgr.getBatcher() != null) {
mgr.getBatcher()
.offerStatementExecution(
StatementExecution.newBuilder()
.setStatementType(statementType)
.setQueryApiType(apiType)
.setStatus(status)
.setErrorCode(errorCode)
.setCount(1)
.setDuration(toDurationBucketMs(durationMs))
.build());
}
});
}

static void recordFeatureUsage(DriverFeature feature, String customFeatureName) {
runSafely(
() -> {
TelemetryManager mgr = instance;
if (mgr != null && mgr.getBatcher() != null) {
mgr.getBatcher()
.offerFeatureUsage(
FeatureUsage.newBuilder()
.setDriverFeature(feature)
.setCustomFeatureName(customFeatureName == null ? "" : customFeatureName)
.setCount(1)
.build());
}
});
}
}
56 changes: 41 additions & 15 deletions java-bigquery-jdbc/src/main/proto/google/cloud/bigquery/jdbc/telemetry/v1/telemetry.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ enum AuthenticationType {
// Aggregated metrics for connection attempts.
message ConnectionAttempt {
Status status = 1;
string error_code = 2;
int32 error_code = 2;
AuthenticationType auth_type = 3;
int64 count = 4;
}
Expand All @@ -89,18 +89,44 @@ enum StatementType {
STATEMENT_TYPE_INSERT = 2;
STATEMENT_TYPE_UPDATE = 3;
STATEMENT_TYPE_DELETE = 4;
STATEMENT_TYPE_MERGE = 5;
STATEMENT_TYPE_CREATE_TABLE = 6;
STATEMENT_TYPE_CREATE_MODEL = 7;
STATEMENT_TYPE_CREATE_VIEW = 8;
STATEMENT_TYPE_DROP_TABLE = 9;
STATEMENT_TYPE_DROP_VIEW = 10;
STATEMENT_TYPE_ALTER_TABLE = 11;
STATEMENT_TYPE_ALTER_VIEW = 12;
STATEMENT_TYPE_SCRIPT = 13;
STATEMENT_TYPE_CALL = 14;
STATEMENT_TYPE_EXPLAIN = 15;
STATEMENT_TYPE_OTHER = 16;
STATEMENT_TYPE_CREATE_EXTERNAL_TABLE = 5;
STATEMENT_TYPE_CREATE_FUNCTION = 6;
STATEMENT_TYPE_CREATE_MATERIALIZED_VIEW = 7;
STATEMENT_TYPE_CREATE_MODEL = 8;
STATEMENT_TYPE_CREATE_PROCEDURE = 9;
STATEMENT_TYPE_CREATE_ROW_ACCESS_POLICY = 10;
STATEMENT_TYPE_CREATE_SCHEMA = 11;
STATEMENT_TYPE_CREATE_SEARCH_INDEX = 12;
STATEMENT_TYPE_CREATE_SNAPSHOT_TABLE = 13;
STATEMENT_TYPE_CREATE_TABLE = 14;
STATEMENT_TYPE_CREATE_TABLE_AS_SELECT = 15;
STATEMENT_TYPE_CREATE_TABLE_FUNCTION = 16;
STATEMENT_TYPE_CREATE_VIEW = 17;
STATEMENT_TYPE_DROP_EXTERNAL_TABLE = 18;
STATEMENT_TYPE_DROP_FUNCTION = 19;
STATEMENT_TYPE_DROP_MATERIALIZED_VIEW = 20;
STATEMENT_TYPE_DROP_MODEL = 21;
STATEMENT_TYPE_DROP_PROCEDURE = 22;
STATEMENT_TYPE_DROP_ROW_ACCESS_POLICY = 23;
STATEMENT_TYPE_DROP_SCHEMA = 24;
STATEMENT_TYPE_DROP_SEARCH_INDEX = 25;
STATEMENT_TYPE_DROP_SNAPSHOT_TABLE = 26;
STATEMENT_TYPE_DROP_TABLE = 27;
STATEMENT_TYPE_DROP_TABLE_FUNCTION = 28;
STATEMENT_TYPE_DROP_VIEW = 29;
STATEMENT_TYPE_ALTER_MATERIALIZED_VIEW = 30;
STATEMENT_TYPE_ALTER_SCHEMA = 31;
STATEMENT_TYPE_ALTER_TABLE = 32;
STATEMENT_TYPE_ALTER_VIEW = 33;
STATEMENT_TYPE_CALL = 34;
STATEMENT_TYPE_EXPLAIN = 35;
STATEMENT_TYPE_EXPORT_DATA = 36;
STATEMENT_TYPE_EXPORT_MODEL = 37;
STATEMENT_TYPE_LOAD_DATA = 38;
STATEMENT_TYPE_MERGE = 39;
STATEMENT_TYPE_SCRIPT = 40;
STATEMENT_TYPE_TRUNCATE_TABLE = 41;
STATEMENT_TYPE_OTHER = 42;
}

// API substrate used during query execution (REST API vs Read API vs Write API vs Jobless Query).
Expand All @@ -117,7 +143,7 @@ message StatementExecution {
StatementType statement_type = 1;
QueryApiType query_api_type = 2;
Status status = 3;
string error_code = 4;
int32 error_code = 4;
int64 count = 5;
DurationHistogram duration = 6;
}
Expand All @@ -139,7 +165,7 @@ message DurationHistogram {

// Aggregated error counts grouped by error code, SQL state, and method.
message ErrorMetric {
string error_code = 1;
int32 error_code = 1;
string error_xdbc_code = 2;
string method_name = 3;
int64 count = 4;
Expand Down
3 changes: 1 addition & 2 deletions ...-jdbc/src/test/java/com/google/cloud/bigquery/jdbc/telemetry/v1/TelemetryBatcherTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,7 @@ public LowLevelHttpResponse execute() {
ConnectionAttempt.newBuilder().setStatus(Status.STATUS_SUCCESS).build());
batcher.offerStatementExecution(
StatementExecution.newBuilder().setStatus(Status.STATUS_SUCCESS).build());
batcher.offerErrorMetric(
ErrorMetric.newBuilder().setErrorCode("ERR_001").setCount(1).build());
batcher.offerErrorMetric(ErrorMetric.newBuilder().setErrorCode(1).setCount(1).build());
batcher.offerFeatureUsage(
FeatureUsage.newBuilder().setDriverFeature(DriverFeature.DRIVER_FEATURE_CUSTOM).build());

Expand Down
2 changes: 1 addition & 1 deletion ...src/test/java/com/google/cloud/bigquery/jdbc/telemetry/v1/TelemetryConfigurationTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ public void testDefaultValues() {
TelemetryConfiguration config = TelemetryConfiguration.newBuilder().build();

assertTrue(config.isEnabled());
assertEquals(-1, config.getLogSource());
assertEquals(3071, config.getLogSource());
assertEquals("https://play.googleapis.com/log", config.getEndpointUrl());
assertEquals(300_000L, config.getUploadIntervalMs());
assertEquals(5000, config.getBatchSizeThreshold());
Expand Down
Loading
Loading