ci: speed up backend image build (step 1)#4685
Merged
Merged
Conversation
Move the backend build cache from the GitHub Actions cache backend to the ECR registry, and drop the unused QEMU setup step. The 12-minute backend builds were dominated by exporting the build cache to GitHub Actions (~7 min, mode=max re-uploading every layer of every stage to a throttled backend). Pointing the cache at the same ECR repo lets the image push and cache export share one content-addressed store, so the heavy .venv layer uploads once instead of twice, and ECR is not rate-limited like the GHA cache backend. image-manifest=true,oci-mediatypes=true are required: ECR rejects buildkit's legacy cache-manifest format. mode=max is kept on purpose — this is a multi-stage build and min would stop caching the intermediate uv sync / pnpm build stages, which would re-run on every ephemeral CI runner. QEMU is removed because the runner is ubuntu-24.04-arm building linux/arm64 natively, so no emulation is needed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Switches backend Docker build cache from GitHub Actions to ECR registry and removes the now-redundant QEMU setup step since the runner is natively ARM. No issues found. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two changes to
.github/workflows/build-backend.yml:pycon-backendrepo,:buildcachetag).ubuntu-24.04-armbuildinglinux/arm64natively, no emulation needed.Why
The backend image build was taking ~12 min. Almost none of it was compiling — it was moving big layers around. Breakdown of the
Build and pushstep (~11m55s):uv sync, schema,.venvcopy, collectstatic)mode=max)The dominant cost was exporting the cache to the GHA backend:
mode=maxre-uploads every layer of every stage to a separate, rate-limited store. The heavy.venvlayer (multi-GB —sentence-transformerspulls in torch + transformers) got uploaded twice: once to ECR as the image, once to GHA as cache.Pointing the cache at the same ECR repo lets the image push and cache export share one content-addressed store, so shared layers upload once, and ECR isn't throttled like the GHA cache backend. Expected saving ~3–4 min/build.
Notes
image-manifest=true,oci-mediatypes=trueare required — ECR rejects buildkit's legacy cache-manifest format.mode=maxkept on purpose: this is a multi-stage build;minwould stop caching the intermediateuv sync/pnpm buildstages, which would then re-run on every ephemeral CI runner.:buildcachetag doesn't exist yet → cache miss, builds fresh, creates it. Speedup lands on run Login / Authentication #2+.This is step 1
This is the first step. The real fix is shrinking the image: the multi-GB
.venv(torch/transformers viasentence-transformers, plus scikit-learn/nltk) bloats every phase —.venvcopy, image export, image push, and cache export. Step 2 will move the ML stuff out (CPU-only torch / a separate ML worker image) so the web image stays small. That's the change that cuts every phase regardless of cache backend; this PR just stops the bleeding on the cache export first.🤖 Generated with Claude Code