CLDSRV-872: store checksum in object metadata by leif-scality · Pull Request #6114 · scality/cloudserver

leif-scality · 2026-03-17T15:49:59Z

Store checksum for objects uploaded with PutObject method
Return the object checksum in the response headers of PutObject and UploadPart
Storing the checksum of a part uploaded with UploadPart will be done in another PR

bert-e · 2026-03-17T15:50:04Z

Hello leif-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-03-17T15:50:11Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-872 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.4.0

Please check the Fix Version/s of CLDSRV-872, or the target
branch of this pull request.

codecov · 2026-03-17T15:54:30Z

Codecov Report

❌ Patch coverage is 90.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.76%. Comparing base (71a178e) to head (8f9c848).
⚠️ Report is 6 commits behind head on development/9.4.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
lib/api/apiUtils/object/createAndStoreObject.js	82.35%	3 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/api/apiUtils/object/storeObject.js	`100.00% <100.00%> (ø)`
lib/api/objectPut.js	`94.95% <100.00%> (+0.13%)`	⬆️
lib/api/objectPutPart.js	`87.09% <100.00%> (-0.82%)`	⬇️
lib/routes/routeBackbeat.js	`76.86% <ø> (+0.54%)`	⬆️
lib/services.js	`87.07% <100.00%> (+0.10%)`	⬆️
lib/api/apiUtils/object/createAndStoreObject.js	`85.52% <82.35%> (+0.12%)`	⬆️

... and 2 files with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.4    #6114      +/-   ##
===================================================
+ Coverage            84.69%   84.76%   +0.07%     
===================================================
  Files                  207      207              
  Lines                13543    13569      +26     
===================================================
+ Hits                 11470    11502      +32     
+ Misses                2073     2067       -6

Flag	Coverage Δ
file-ft-tests	`68.60% <90.00%> (+0.03%)`	⬆️
kmip-ft-tests	`28.11% <36.66%> (+0.01%)`	⬆️
mongo-v0-ft-tests	`69.82% <90.00%> (-0.03%)`	⬇️
mongo-v1-ft-tests	`69.85% <90.00%> (+0.11%)`	⬆️
multiple-backend	`36.45% <43.33%> (+0.12%)`	⬆️
sur-tests	`35.61% <36.66%> (-0.79%)`	⬇️
sur-tests-inflights	`37.42% <50.00%> (-0.01%)`	⬇️
unit	`70.66% <86.66%> (+0.03%)`	⬆️
utapi-v2-tests	`34.63% <56.66%> (+0.09%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

package.json

lib/api/apiUtils/object/storeObject.js

claude · 2026-03-17T16:01:31Z

Arsenal dependency is pinned to a branch instead of a tag. Must be pinned to a release tag before merging.
- Change arsenal in package.json to point to a tagged version.
Verify ObjectMDChecksum constructor argument names match the field names used in storeObject.js checkHashMatchMD5 (algorithm, value, type). The unit tests assert checksumAlgorithm, checksumValue, checksumType on the stored metadata. If ObjectMDChecksum maps these internally it is fine, but worth confirming.
In storeObject.js dataStore, the onStreamError handler (line 91) and doValidate via once('finish', ...) (line 125) can both fire if the stream emits both error and finish. jsutil.once(cb) prevents a double callback, but both paths would call batchDelete independently. Consider removing the opposing listener when one fires to avoid a redundant delete call.

Review by Claude Code

package.json

lib/api/apiUtils/object/storeObject.js

claude · 2026-03-17T16:50:33Z

Review summary: (1) Arsenal dependency pinned to a branch instead of a tag - must update to a released tag. (2) checkHashMatchMD5 JSDoc missing the new checksumStream parameter. (3) dataStore callback returns a 4th checksum arg but objectPutPart and routeBackbeat callers ignore it. -- Review by Claude Code

package.json

claude · 2026-03-17T17:26:40Z

Arsenal dependency is pinned to a branch instead of a tag, which makes builds non-reproducible. Pin to a released tag before merging.

Review by Claude Code

package.json

claude · 2026-03-17T19:07:51Z

Arsenal dependency is pinned to a feature branch instead of a release tag. Pin to a tag before merging.

Review by Claude Code

package.json

claude · 2026-03-17T19:41:05Z

Arsenal is pinned to a feature branch instead of a release tag. This must be updated to a tagged version before merging to ensure reproducible builds.

Review by Claude Code

package.json

lib/routes/routeBackbeat.js

claude · 2026-03-18T12:01:21Z

Arsenal dependency is pinned to a branch instead of a tag. Must be updated to a released tag before merge.
Misleading comment placement in routeBackbeat.js (two places): the checksum 4th-arg comment sits between dataStore positional args. Move it above the callback.

Review by Claude Code

package.json

lib/api/apiUtils/object/createAndStoreObject.js

claude · 2026-03-18T12:12:49Z

Arsenal dependency is pinned to a feature branch instead of a release tag. Must be updated to a tagged version before merging.
- Pin to the release tag that includes the ObjectMDChecksum changes.
zeroSizeBodyChecksumCheck mixes callbacks with Promises, creating a double-callback risk: if callback throws inside .then(), the .catch() will invoke callback(err) a second time.
- Use process.nextTick inside the .then() and .catch() to break out of the promise chain, or convert to async/await with try/catch.

Review by Claude Code

package.json

claude · 2026-03-18T12:21:25Z

Arsenal dependency is pinned to a branch instead of a tag. This must be updated to a release tag before merging for reproducible builds.

Review by Claude Code

lib/api/apiUtils/object/createAndStoreObject.js

tests/functional/raw-node/test/routes/routeMetadata.js

fredmnl · 2026-03-25T09:48:27Z

tests/functional/raw-node/test/checksumPutObjectUploadPart.js

+                        assert.strictEqual(res.headers['x-amz-checksum-crc64nvme'], crc64nvmeOfTestContent2,
+                            `expected x-amz-checksum-crc64nvme: ${crc64nvmeOfTestContent2}`);
+                    }
+                    done();


Should we fail this test if the checksum header is absent?

Same for all tests below

UploadPart now return the checksum of the part, so I removed the condition and always require a checksum

fredmnl · 2026-03-25T09:57:08Z

tests/functional/raw-node/test/checksumPutObjectUploadPart.js

+        crc32c: 'AAAAAA==',
+        sha1: '2jmj7l5rSw0yVb/vlWAYkK/YBwk=',
+        sha256: '47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=',
+        crc64nvme: null, // filled in before hook (async)


nit: Not sure why we're not writing this one down as well

hardcoded now

tests/functional/raw-node/test/routes/routeMetadata.js

tests/functional/raw-node/test/checksumPutObjectUploadPart.js

claude · 2026-03-26T17:09:01Z

Test names using it() should start with "should" (convention enforced across the project).
- tests/functional/raw-node/test/routes/routeMetadata.js: e.g., "stores sha256 checksum..." -> "should store sha256 checksum..."
- tests/functional/raw-node/test/checksumPutObjectUploadPart.js: e.g., "returns x-amz-checksum-..." -> "should return x-amz-checksum-..."

Review by Claude Code

…bject

claude · 2026-03-27T15:03:54Z

tests/functional/raw-node/test/checksumPutObjectUploadPart.js

+        });
+    });
+
+    const checksumAlgos = [


Test names using it() should start with "should" per project conventions. For example: "should return x-amz-checksum-... response header with correct value". This applies to several new it()/itSkipIfAWS() tests in this file (lines 761, 774, 793, 813, 820, etc.).

— Claude Code

claude · 2026-03-27T15:03:59Z

Test naming: several new it()/itSkipIfAWS() test names in checksumPutObjectUploadPart.js do not start with "should" (project convention).
- Prefix test descriptions with "should" (e.g. "should return x-amz-checksum-... response header").

Review by Claude Code

claude · 2026-03-27T15:13:51Z

LGTM

Review by Claude Code

leif-scality · 2026-03-27T15:37:17Z

ping

bert-e · 2026-03-27T15:37:26Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

leif-scality · 2026-03-27T15:38:34Z

/approve

bert-e · 2026-03-27T15:38:45Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/9.4

The following branches have NOT changed:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2
development/9.3

Please check the status of the associated issue CLDSRV-872.

Goodbye leif-scality.

The following options are set: approve

leif-scality changed the base branch from development/9.4 to improvement/CLDSRV-863-checksums-put-object-part March 17, 2026 15:51

claude bot reviewed Mar 17, 2026

View reviewed changes

package.json Outdated Show resolved Hide resolved

claude bot reviewed Mar 17, 2026

View reviewed changes

lib/api/apiUtils/object/storeObject.js Show resolved Hide resolved

leif-scality force-pushed the improvement/CLDSRV-872-store-checksum-in-object-metadata branch from 5d6c434 to 1b5a063 Compare March 17, 2026 16:48

claude bot reviewed Mar 17, 2026

View reviewed changes

package.json Outdated Show resolved Hide resolved

claude bot reviewed Mar 17, 2026

View reviewed changes

lib/api/apiUtils/object/storeObject.js Show resolved Hide resolved

claude bot reviewed Mar 17, 2026

View reviewed changes

lib/api/apiUtils/object/storeObject.js Show resolved Hide resolved

leif-scality force-pushed the improvement/CLDSRV-872-store-checksum-in-object-metadata branch from 1b5a063 to b80b2ad Compare March 17, 2026 17:24

claude bot reviewed Mar 17, 2026

View reviewed changes

package.json Outdated Show resolved Hide resolved

claude bot reviewed Mar 17, 2026

View reviewed changes

package.json Outdated Show resolved Hide resolved

claude bot reviewed Mar 17, 2026

View reviewed changes

package.json Outdated Show resolved Hide resolved

claude bot reviewed Mar 18, 2026

View reviewed changes

package.json Outdated Show resolved Hide resolved

claude bot reviewed Mar 18, 2026

View reviewed changes

lib/routes/routeBackbeat.js Outdated Show resolved Hide resolved

claude bot reviewed Mar 18, 2026

View reviewed changes

package.json Outdated Show resolved Hide resolved

claude bot reviewed Mar 18, 2026

View reviewed changes

lib/api/apiUtils/object/createAndStoreObject.js Outdated Show resolved Hide resolved

claude bot reviewed Mar 18, 2026

View reviewed changes

package.json Outdated Show resolved Hide resolved

BourgoisMickael reviewed Mar 19, 2026

View reviewed changes

lib/api/apiUtils/object/createAndStoreObject.js Outdated Show resolved Hide resolved

leif-scality force-pushed the improvement/CLDSRV-863-checksums-put-object-part branch from a284c9f to dade07b Compare March 20, 2026 09:54

leif-scality force-pushed the improvement/CLDSRV-872-store-checksum-in-object-metadata branch from 180d940 to 7d130cc Compare March 20, 2026 15:45

BourgoisMickael reviewed Mar 23, 2026

View reviewed changes

tests/functional/raw-node/test/routes/routeMetadata.js Outdated Show resolved Hide resolved

BourgoisMickael approved these changes Mar 23, 2026

View reviewed changes

fredmnl self-requested a review March 23, 2026 17:28

fredmnl approved these changes Mar 25, 2026

View reviewed changes

leif-scality force-pushed the improvement/CLDSRV-863-checksums-put-object-part branch 4 times, most recently from 1167cf2 to 71a178e Compare March 26, 2026 14:09

Base automatically changed from improvement/CLDSRV-863-checksums-put-object-part to development/9.4 March 26, 2026 15:19

leif-scality force-pushed the improvement/CLDSRV-872-store-checksum-in-object-metadata branch from 8e51413 to b78f50e Compare March 26, 2026 17:04

claude bot reviewed Mar 26, 2026

View reviewed changes

tests/functional/raw-node/test/routes/routeMetadata.js Show resolved Hide resolved

claude bot reviewed Mar 26, 2026

View reviewed changes

tests/functional/raw-node/test/checksumPutObjectUploadPart.js Show resolved Hide resolved

CLDSRV-872: store checksum in metadata for objects uploaded with PutO…

02d5b38

…bject

leif-scality force-pushed the improvement/CLDSRV-872-store-checksum-in-object-metadata branch from b78f50e to fea5f58 Compare March 27, 2026 15:01

claude bot reviewed Mar 27, 2026

View reviewed changes

leif-scality added 5 commits March 27, 2026 16:11

CLDSRV-872: return stored checksum in PutObject response

0edd600

CLDSRV-872: handle empty object checksum

41c8962

CLDSRV-872: fix failing test, checksum lost when restoring object

1319d2a

CLDSRV-872: test checksums are respected in backbeat routes

798f8c1

CLDSRV-872: add missing jsdoc

8f9c848

leif-scality force-pushed the improvement/CLDSRV-872-store-checksum-in-object-metadata branch from fea5f58 to 8f9c848 Compare March 27, 2026 15:11

bert-e merged commit 8f9c848 into development/9.4 Mar 27, 2026
31 checks passed

bert-e deleted the improvement/CLDSRV-872-store-checksum-in-object-metadata branch March 27, 2026 15:38

Conversation

leif-scality commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bert-e commented Mar 17, 2026

Hello leif-scality,

Uh oh!

bert-e commented Mar 17, 2026

Incorrect fix version

Uh oh!

codecov bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

claude bot commented Mar 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

claude bot commented Mar 17, 2026

Uh oh!

Uh oh!

claude bot commented Mar 17, 2026

Uh oh!

Uh oh!

claude bot commented Mar 17, 2026

Uh oh!

Uh oh!

claude bot commented Mar 17, 2026

Uh oh!

Uh oh!

Uh oh!

claude bot commented Mar 18, 2026

Uh oh!

Uh oh!

Uh oh!

claude bot commented Mar 18, 2026

Uh oh!

Uh oh!

claude bot commented Mar 18, 2026

Uh oh!

Uh oh!

Uh oh!

fredmnl Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

fredmnl Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

leif-scality Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

fredmnl Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

leif-scality Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

claude bot commented Mar 26, 2026

Uh oh!

claude bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

claude bot commented Mar 27, 2026

Uh oh!

claude bot commented Mar 27, 2026

Uh oh!

leif-scality commented Mar 27, 2026

Uh oh!

bert-e commented Mar 27, 2026

Waiting for approval

Uh oh!

leif-scality commented Mar 27, 2026

Uh oh!

bert-e commented Mar 27, 2026

Uh oh!

leif-scality commented Mar 17, 2026 •

edited

Loading

codecov bot commented Mar 17, 2026 •

edited

Loading