Edge-cache object bytes: chunk-aligned ranged-GET caching via worker-controlled Cache API

[alukach]

Status

Proposed

Context

Object bytes are never edge-cached: every GET through the proxy returns cf-cache-status: DYNAMIC, so each request pays the full two-hop fetch (client→edge, edge→origin→edge→client) even for immutable, public objects.

Internal benchmarking of the public endpoints shows the cost is per-request latency, not bandwidth: on a warm connection, a ranged GET through the proxy takes roughly 2–3x as long to first byte as the same request direct to the origin bucket, while bulk throughput within a request is at parity. Cloud-native workloads feel this acutely — a windowed parquet read (GeoPandas bbox=, DuckDB bbox + ST_Intersects) is a mostly-serial chain of 6–10 small ranged GETs (HEAD, footer, metadata, row groups), so the per-request tax compounds to a consistent ~2x end-to-end slowdown.

This has product cost: dataset READMEs now steer users away from data.source.coop toward direct s3:// URIs (e.g. humane-intelligence/bias-bounty-mapping-equity-challenge, which documents the proxy as "substantially slower"). Direct-bucket escape hatches only exist for public AWS-hosted products — Azure/GCS backends and private products have no alternative to the proxy, so proxy latency matters most exactly where users can't route around it.

A warm edge hit returns in ~one client-to-PoP RTT, which is less than a round trip to the origin region — with edge caching, the proxy flips from ~2x slower to strictly faster than direct bucket access for windowed reads.

Constraints

1. Private products require per-request authorization (subject resolution / SigV4); cached bytes must never bypass it.
2. Per-product analytics (Analytics Engine, src/analytics.rs) must keep seeing every request.
3. Ranged GETs dominate the workload; the Cache API cannot store 206 responses.
4. Objects can be large (multi-GB) and can be overwritten.

Decision (proposed)

Worker-controlled, chunk-aligned caching of object bytes via the Cache API — the same mechanism already used for Source API metadata (src/source_api/cache.rs), extended to data.

Per GET, the request flow keeps its existing shape; only the backend fetch changes:

1. Resolve product + authorize (unchanged — runs on every request, hit or miss).
2. Log analytics (unchanged; add a cache_hit dimension).
3. Serve bytes through a chunk cache:
   • Normalize the requested range to fixed-size blocks (e.g. 4 MiB, aligned).
   • cache.match each block; fetch missing blocks from the backend as ranged GETs and cache.put them as 200 responses under a synthetic key: canonical {account}/{product}/{path} + chunk index + object ETag. (ETag in the key makes overwrites self-invalidating — no purge machinery.)
   • Assemble the client's exact range from blocks; respond 206.
4. Large ranges (e.g. > 32 MiB) bypass the chunk cache and stream directly, bounding worker subrequest count and memory.

Initial policy: cache public products only. Authorization runs on every request regardless, so caching private objects is safe in principle (the cache is a byte store; the worker gates access), but public-only is trivial to audit and covers the workloads users benchmark.

The cache key must never include auth material (signatures, tokens, subject) — key = content identity only.

Alternatives considered

• Read-through Workers Caching (zone-level): rejected. On a HIT the worker is not invoked (breaks constraints 1 and 2), and for Range requests Cloudflare strips the Range header and requests the full body from the worker — a first windowed read of a multi-GB file would trigger a full-object origin fetch, and objects over the cacheable size limit never cache.
• Cache client ranges verbatim: rejected. The Cache API won't store 206s, and overlapping-but-unequal ranges (the normal parquet pattern) would never reuse each other. Chunk alignment manufactures the reuse.
• Cache whole objects on first touch: rejected. Multi-GB objects blow size limits and turn a ~100 ms windowed read into a multi-GB origin fetch.
• Status quo (steer users to s3://): what READMEs are already doing on our behalf. Doesn't exist for private products or non-AWS backends.

Consequences

• Warm windowed reads drop to ~one PoP RTT per request; parquet footer/metadata chunks are re-read on every file open by every client, so they cache extremely hot even for a single user iterating.
• Reduced origin egress and request count.
• Cache API is per-PoP (no tiered cache): cross-user wins only materialize within a PoP; metadata chunks still win everywhere.
• No request collapsing: a cold-object burst fans out to origin once per request. Same class of problem as Federated credential cache is per-isolate; consider cross-instance caching (KV/DO) + single-flight #148 (single-flight); acceptable to ship without, worth sharing a solution.
• Worker does more subrequests per client request (one per missing chunk) and assembles responses — CPU/memory cost bounded by the chunk size and the large-range bypass.
• Analytics gains a hit/miss dimension, which also answers "is this working" empirically after launch.

Open questions

• Chunk size (4 MiB strawman): balances origin request count vs waste on small reads (parquet footers are KBs — consider a smaller first/last-chunk size or a metadata-tail heuristic).
• Bypass threshold for large ranges.
• Whether HEAD responses and LIST results should ride the same cache (likely yes for HEAD, with short TTL).
• Extending to private products later (per-request authz already makes it safe; needs a careful look at cache-key hygiene first).

🤖 Generated with Claude Code