Skip to content

ci: use project bucket for assets COMPASS-10105#7682

Merged
mabaasit merged 19 commits intomainfrom
COMPASS-10105-use-project-buckets
Jan 19, 2026
Merged

ci: use project bucket for assets COMPASS-10105#7682
mabaasit merged 19 commits intomainfrom
COMPASS-10105-use-project-buckets

Conversation

@mabaasit
Copy link
Copy Markdown
Collaborator

@mabaasit mabaasit commented Jan 5, 2026

We are moving away from storing artifacts in mciuploads to project specific S3 buckets. DEVPROD-25311 documents how to create these buckets.

Description

Checklist

  • New tests and/or benchmarks are included
  • Documentation is changed or added
  • If this change updates the UI, screenshots/videos are added and a design review is requested
  • If this change could impact the load on the MongoDB cluster, please describe the expected and worst case impact
  • I have signed the MongoDB Contributor License Agreement (https://www.mongodb.com/legal/contributor-agreement)

Motivation and Context

  • Bugfix
  • New feature
  • Dependency update
  • Misc

Open Questions

Dependents

Types of changes

  • Backport Needed
  • Patch (non-breaking change which fixes an issue)
  • Minor (non-breaking change which adds functionality)
  • Major (fix or feature that would cause existing functionality to change)

Comment thread .evergreen/functions.yml Outdated
- command: s3.put
params:
<<: *save-artifact-params-public
<<: *save-artifact-params-private
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these needs to be -private instead of -public? With -private, we can't pass URLs to these builds to people anymore (esp. customers outside of the org), and it makes bulk downloads quite a bit more annoying (which I've had to do for SSDLC artifact compilation from time to time)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am checking this with the team if these can be public.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, backstage created s3 buckets with public access disabled and that's why I was not able to upload assets with public access which forced me to use private. DEVPROD will change this and I'll revert this bit.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, so DEVPROD wants us to keep the assets with private access. Regarding ssdlc report, would it help if we prioritize COMPASS-8030?

Comment thread .evergreen/build-dev-release-info.sh Outdated
)

URL="https://mciuploads.s3.amazonaws.com/${EVERGREEN_PROJECT}/compass/dev/$1"
URL="https://s3.amazonaws.com/downloads.10gen.com/compass/dev/$1"
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: Update update-server with this url

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mabaasit
Copy link
Copy Markdown
Collaborator Author

In order to avoid having aws credentials for update server, compass (gha), DEVPROD-26085 helped us with compass-dev bucket that has lower retention and is accessible via cdn. Artifacts are stored with private access but publicly accessible via cdn.

@mabaasit mabaasit marked this pull request as ready for review January 13, 2026 21:12
@mabaasit mabaasit requested a review from a team as a code owner January 13, 2026 21:12
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates artifact storage from the generic mciuploads S3 bucket to a project-specific cdn-origin-compass-dev bucket, aligning with MongoDB's infrastructure modernization efforts.

Changes:

  • Updated S3 bucket references from mciuploads to cdn-origin-compass-dev with role-based authentication
  • Removed EVERGREEN_BUCKET_NAME environment variable dependency across the codebase
  • Introduced build attestations module to systematically manage artifact metadata and upload paths

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
.evergreen/functions.yml Updated artifact storage parameters to use project-specific bucket with role ARN authentication
packages/hadron-build/lib/download-center.js Modified download URL to use new CDN path structure
packages/hadron-build/lib/build-attestations.js New module for managing build attestation file paths and metadata
packages/hadron-build/commands/upload.js Integrated attestations into the upload workflow
packages/hadron-build/commands/download.js Added attestation download support
packages/compass-smoke-tests/src/test-subject.ts Updated download URL to new CDN location
packages/compass-smoke-tests/src/dispatch.ts Removed bucket name parameter from workflow dispatch
packages/compass-smoke-tests/src/context.ts Removed bucket name from context type
packages/compass-smoke-tests/src/cli.ts Removed bucket name CLI option and validation
.github/workflows/test-installers.yml Removed bucket name workflow input
.evergreen/print-debug-info.sh Removed bucket name from debug output
.evergreen/buildvariants-and-tasks.yml Updated first-party dependency file path structure
.evergreen/buildvariants-and-tasks.in.yml Updated first-party dependency filename template
.evergreen/build-dev-release-info.sh Updated release info URL to use new CDN location
packages/compass-smoke-tests/package.json Reordered dependencies alphabetically
packages/hadron-build/test/build-attestations.spec.js Added comprehensive test coverage for new attestations module

Comment thread packages/hadron-build/lib/build-attestations.js Outdated
Comment thread packages/hadron-build/commands/download.js
Comment thread .evergreen/build-dev-release-info.sh
Copy link
Copy Markdown
Collaborator

@addaleax addaleax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I'd suggest holding off on merging this until next week (after the AI assistant release + any potential hotfixes afterwards), if that's possible

Comment thread packages/hadron-build/lib/build-attestations.js Outdated
@mabaasit mabaasit merged commit c7d9a5e into main Jan 19, 2026
56 of 57 checks passed
@mabaasit mabaasit deleted the COMPASS-10105-use-project-buckets branch January 19, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants