Skip to content

[ci] Refactor CI: merge-queue-gated pr-check + release pipeline#885

Merged
chuandew merged 1 commit into
dingodb:mainfrom
chuandew:feat/ci-v5-merge-queue
May 15, 2026
Merged

[ci] Refactor CI: merge-queue-gated pr-check + release pipeline#885
chuandew merged 1 commit into
dingodb:mainfrom
chuandew:feat/ci-v5-merge-queue

Conversation

@chuandew
Copy link
Copy Markdown
Member

Summary

Refactor dingofs CI/CD to a streamlined 3-workflow architecture gated by GitHub Merge Queue, replacing the current ci-build.yml + wheels.yml + upload_package.yml + docker-publish.yml setup.

Workflows after this PR

File Trigger Role
.github/workflows/pr-check.yml pull_request + merge_group PR gate: unit-testbuilde2e (3 status checks)
.github/workflows/release.yml push: main / tags:v* Release pipeline: builddocker-publish + wheelspypi-publish (tag only)
.github/workflows/_release-build.yml workflow_call: only Reusable build engine, called by pr-check + release

Key design choices

  • Merge Queue gating (org-only feature): PR check runs at PR push time + at queue dequeue time with rebased SHA. Guarantees every commit on main has been tested against its precise merge state.
  • Release doesn't re-test unit/build/e2e: Trusts merge queue's gate, only builds artifacts + publishes (saves ~15 min per merge).
  • Single-use sub-workflows inlined: wheels / docker-publish / pypi-publish / unit-test / e2e-smoke were all single-callsite — inlined to their unique caller. Only _release-build.yml retained as reusable (called by both pr-check + release).
  • Dependency strategy: dingo-store image + dingocli use `:latest` on main (in-org, follows upstream); minio image + GHA actions pin sha256/commit (third-party, supply chain). Release branches pin everything for reproducibility — see `.github/README.md` §5.

Maintainer one-time setup required

Before merging, please configure (full instructions in `.github/README.md`):

Secrets (Settings → Secrets and variables → Actions)

  • `DOCKERHUB_USERNAME` / `DOCKERHUB_TOKEN`
  • `PYPI_API_TOKEN`

Branch Protection (Settings → Branches → main)

  • Require pull request before merging, approvals: 1
  • Require status checks: `unit-test`, `build / release-build`, `e2e`
  • Require branches to be up to date before merging
  • Require merge queue (REBASE, ALLGREEN, build concurrency 5, min 1 / max 5 / wait 5min)
  • Require linear history
  • Do not allow bypassing the above

Default Merge Method (Settings → General): Rebase and merge

Verification

  • Local simulate: `bash .github/scripts/simulate-locally.sh` passes 119/119 pytest in ~196s on 9950X
  • Personal fork PR Check: chuandew/dingofs PR Check workflow runs green on this branch (validates pr-check.yml gating)
  • Cannot validate on personal fork: `merge_group` event + queue rebase + race protection — Merge Queue is GitHub org-only feature, needs dingodb org admin to enable and validate

Related validation PR on fork: chuandew#1

Test plan (after merging + maintainer setup)

  • Trivial follow-up PR triggers 3 status checks on pr-check.yml
  • Merge button shows "Add to merge queue" when all green
  • Queue dequeue triggers pr-check.yml with `merge_group` event (rebased SHA)
  • After merge, release.yml triggers on push:main → docker hub gets new `latest` + short-sha tags
  • Push `v0.x.0` test tag → release.yml runs full pipeline → DockerHub + PyPI get new artifacts

@chuandew chuandew force-pushed the feat/ci-v5-merge-queue branch 2 times, most recently from 95d3910 to e1895ae Compare May 15, 2026 02:10
Copy link
Copy Markdown
Contributor

@Wine93 Wine93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@chuandew chuandew enabled auto-merge May 15, 2026 03:56
@chuandew chuandew force-pushed the feat/ci-v5-merge-queue branch from e1895ae to 905af3a Compare May 15, 2026 03:58
Copy link
Copy Markdown
Contributor

@rock-git rock-git left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chuandew chuandew added this pull request to the merge queue May 15, 2026
Merged via the queue into dingodb:main with commit 8c7a6ee May 15, 2026
3 checks passed
@chuandew chuandew deleted the feat/ci-v5-merge-queue branch May 15, 2026 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants