Validate Databricks CLI token scopes against SDK configuration

[tejaskochar-db]

Summary

Detects when a cached Databricks CLI token was issued with different OAuth scopes than what the SDK configuration requires, and surfaces an actionable error telling the user how to re-authenticate instead of silently making requests with the wrong scopes.

Mirrors Python SDK PR: databricks/databricks-sdk-py#1286

Why

The databricks auth token CLI command does not accept scopes — it returns whatever token was cached from the last databricks auth login. If a user configures specific scopes in the SDK (e.g. scopes=["sql"], either directly in code or loaded from a CLI profile), but their cached token was issued with different scopes (e.g. all-apis), every API request silently uses the wrong scopes. This is confusing to debug because authentication succeeds — it just grants the wrong permissions.

This is especially likely now that the CLI writes scopes into profiles, meaning users who switch between different scope configurations will hit this without realizing it.

What changed

Behavioral changes

• Scope mismatch error — When scopes are explicitly set in the SDK config and the cached CLI token's JWT scope claim doesn't match, a DatabricksException is raised with a message like:

  Token issued by Databricks CLI has scopes [all-apis] which do not match
  the configured scopes [sql]. Please re-authenticate with the desired scopes
  by running `databricks auth login --host <host> --scopes sql`.
  

• Credential chain fallthrough — When authType is not explicitly set to "databricks-cli" (i.e. we're being tried as part of the default credential chain), a scope mismatch logs a warning and returns null so other providers get a chance. When authType="databricks-cli" is explicitly set, the error propagates immediately.

• offline_access is ignored during comparison — This scope controls refresh token issuance, not API permissions. Its presence or absence on either side does not trigger a mismatch.

• Validation is skipped when scopes are not configured — getScopes() defaults to ["all-apis"] when nothing is set, which would cause false-positive mismatches. Validation only runs when scopes are explicitly set.

Internal changes

• DatabricksCliCredentialsProvider.validateTokenScopes() — Decodes the JWT scope claim (handling both space-delimited strings and JSON arrays), filters out offline_access, and compares against configured scopes using strict set equality.

• DatabricksCliCredentialsProvider.getJwtClaims() — Decodes JWT payload using Base64.getUrlDecoder() (URL-safe base64 per RFC 7519).

• DatabricksConfig.isScopesExplicitlySet() — Returns true when the raw scopes field is non-null and non-empty, covering both programmatic configuration and config file loading via reflection.

How is this tested?

• Parametrized tests covering: exact match, mismatch (error), offline_access filtering (both directions), scope claim as list, non-JWT tokens (skip), no scope claim (skip), and error message format with re-auth command.
• Unit tests for isScopesExplicitlySet() flag behavior.

🤖 Generated with Claude Code

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/sdk-java

Inputs:

• PR number: 689
• Commit SHA: dd6ee4e643cbd1b061185bb997d3681085a2ab4e

Checks will be approved automatically on success.