fix(data-connections): target GCS, align DataProvider with storage_type by alukach · Pull Request #453 · source-cooperative/source.coop

alukach · 2026-07-22T17:53:54Z

Problem

Creating a GCP data connection persisted details.provider: "gcp", but the backend (data.source.coop) addresses Google Cloud Storage as "gcs". The same latent mismatch existed for Azure ("az" vs "azure").

Root cause: DataProvider and a mirror's storage type were two vocabularies bridged by a hand-written STORAGE_TYPE_BY_PROVIDER map duplicated in products.ts and product-mirrors.ts.

Change — align `DataProvider` values to the backend vocabulary

enum DataProvider { S3 = "s3", Azure = "azure" /* was "az" */, GCS = "gcs" /* was GCP="gcp" */ }

Both translation maps deleted; a mirror's storage type is just the connection's provider.
Member GCP→GCS, schema/type Gcp*→Gcs*. Auth type stays gcp_workload_identity (WIF is a GCP-platform feature, not storage). Form label → "Google Cloud Storage".

Migration (legacy `az`/`gcp` rows)

DynamoDB reads bypass Zod, so normalizeConnection in the data-connections client remaps "az"→"azure" / "gcp"→"gcs" on fetchById/listAll. Marked ponytail: — drop after backfill.

Backend compatibility (verified against data.source.coop)

S3 unchanged. Azure accepts both "az" and "azure" → compatible during rollout. GCS: old "gcp" never worked (hit the error arm); "gcs" is accepted. This is the fix, not a regression.
⚠️ Making GCS actually connect needs a separate backend change (enable the multistore gcp feature + a wasm-compatible signing path) — tracked in data.source.coop.

Verification

tsc clean · jest affected suites pass · lint clean on changed files.

🤖 Generated with Claude Code

The GCP data connection persisted `provider: "gcp"`, but the backend addresses Google Cloud Storage as "gcs" — the same latent mismatch existed for Azure ("az" vs "azure"). Root cause: `DataProvider` and a mirror's `storage_type` were two vocabularies bridged by a hand-written `STORAGE_TYPE_BY_PROVIDER` map duplicated in products.ts and product-mirrors.ts. Detangle: align `DataProvider`'s values to the backend storage vocabulary (`Azure="azure"`, `GCS="gcs"`), so a mirror's `storage_type` is just the connection's provider. Both translation maps deleted; `storage_type = details.provider` directly. - Rename member GCP→GCS and schema/type Gcp*→Gcs* (auth type stays `gcp_workload_identity` — WIF is a GCP-platform feature, not storage). - Read-shim in the data-connections DynamoDB client normalizes legacy rows on read ("az"→"azure", "gcp"→"gcs"); mirrors already store the correct storage_type via the old map, so only connection rows need it. - Tests for the shim and for the DataProvider ⊆ storage_type invariant that now backs the cast. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

vercel · 2026-07-22T17:53:59Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
source-cooperative	Ready	Preview, Comment	Jul 22, 2026 6:40pm

claude · 2026-07-22T17:54:15Z

Claude finished @alukach's task in 3m 8s —— View job

Review complete

Gather context (PR diff, changed files)
Review diff for correctness bugs, security issues, best-practice problems
Post findings

❌ Changes requested — see findings below.

fixtures/data-connections.json:58 — still has "provider": "az", which no longer parses: AzureDataConnectionSchema now requires the literal "azure" (src/types/data-connection.ts:233). This fixture is loaded eagerly at module-import time via loadAndValidateJson in src/lib/api/utils.mock.ts:73-76, and src/lib/api/authz.test.ts imports it, so the whole suite fails to run. This is exactly what's failing in this PR's own CI run (CI Checks job, test step — Test suite failed to run: Validation error for file fixtures/data-connections.json: Invalid discriminator value. Expected 's3' | 'azure' | 'gcs'). The "GCP" leg of the fixture already used "gcs" — likely a copy/paste that missed the Azure entry, or the fixture predates a rebase.
```
      "provider": "azure",
```
Fix this →

Everything else in the migration checks out:

DataProvider enum, schemas (GcsDataConnectionSchema), and the two former STORAGE_TYPE_BY_PROVIDER maps are consistently replaced with a direct cast (src/lib/actions/products.ts:232, src/lib/actions/product-mirrors.ts:105). The cast is sound: ProductMirrorSchema.storage_type (src/types/product.ts:12) is z.enum(["s3","azure","gcs","minio","ceph"]), a strict superset of DataProvider's values, and a new test (data-connections.test.ts — DataProvider ⊆ ProductMirror storage_type) guards the assumption.
normalizeConnection (src/lib/clients/database/data-connections.ts:26-31) correctly remaps legacy "az"/"gcp" rows on both fetchById and listAll; update()'s callers all route through an already-normalized fetchById result first, so no write path can re-persist a stale value.
No other stray "az"/"gcp"/Gcp* references remain in src/ outside the intentionally-unchanged gcp_workload_identity auth type (a platform/WIF concept, not storage).

…igning) (#191) ## What Enables the Google Cloud Storage backend in the data proxy so source.coop data connections with `provider: "gcs"` resolve to a real storage client instead of erroring. ## Changes (Cargo.toml / Cargo.lock only) 1. **`gcp` feature** on `multistore` and `multistore-cf-workers` → `BackendType::Gcs` builds a `GoogleCloudStorageBuilder` instead of returning `ConfigError("GCS backend support not enabled")`. 2. **`ring` `wasm32_unknown_unknown_js` feature** (via a direct wasm-target entry): `object_store`'s GCS credential module signs JWTs with `ring::rand::SystemRandom`, whose `SecureRandom` impl on `wasm32-unknown-unknown` is gated behind that ring feature (enables getrandom 0.2's js backend). Without it the Workers build fails to compile: ``` error[E0277]: the trait bound `SystemRandom: SecureRandom` is not satisfied --> object_store-0.13.2/src/gcp/credential.rs ``` ring is transitive (via `object_store`); this only turns the feature on for the wasm target. ## Runtime note source.coop uses **keyless Workload Identity Federation** for GCS, so the RSA private-key signing path isn't exercised at runtime — but the code must compile, which this enables. (getrandom's js backend maps to Web Crypto, which the Workers runtime provides, so it's also safe if ever hit.) ## Pairs with source-cooperative/source.coop#453 — aligns the persisted provider value to `"gcs"` (the string this backend accepts). ## Verification `cargo check` + `cargo clippy -D warnings` + `cargo fmt --check` for **wasm32-unknown-unknown**, and the full `cargo test` suite — all pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

alukach · 2026-07-22T18:48:13Z

Re-split: value alignment (keeping the translation maps) moved to #455, with your form-label edits preserved. This branch's map-deletion moved into the stacked cleanup #456.

source-release-bot Bot added the fix label Jul 22, 2026

This was referenced Jul 22, 2026

refactor(mirrors): drop denormalized ProductMirror.storage_type #454

Closed

fix(data-connections): target GCS, align DataProvider with storage_type #452

Closed

alukach mentioned this pull request Jul 22, 2026

feat(backend): enable GCS backend (multistore gcp + wasm-compatible signing) source-cooperative/data.source.coop#191

Merged

alukach commented Jul 22, 2026

View reviewed changes

Comment thread src/components/features/data-connections/DataConnectionForm.tsx Outdated

Apply suggestion from @alukach

502452a

alukach commented Jul 22, 2026

View reviewed changes

Comment thread src/components/features/data-connections/DataConnectionForm.tsx Outdated

Apply suggestion from @alukach

1d9956c

alukach closed this Jul 22, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(data-connections): target GCS, align DataProvider with storage_type - #453

fix(data-connections): target GCS, align DataProvider with storage_type#453
alukach wants to merge 3 commits into
mainfrom
fix/gcs-provider-alignment

alukach commented Jul 22, 2026

Uh oh!

vercel Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

claude Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

alukach commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

alukach commented Jul 22, 2026

Problem

Change — align DataProvider values to the backend vocabulary

Migration (legacy az/gcp rows)

Backend compatibility (verified against data.source.coop)

Verification

Uh oh!

vercel Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review complete

Uh oh!

Uh oh!

Uh oh!

alukach commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Change — align `DataProvider` values to the backend vocabulary

Migration (legacy `az`/`gcp` rows)

vercel Bot commented Jul 22, 2026 •

edited

Loading

claude Bot commented Jul 22, 2026 •

edited

Loading