Skip to content

fix(cp): apply --destination-region in shouldOverride for --no-clobber (#839)#862

Open
gustcol wants to merge 1 commit intopeak:masterfrom
gustcol:fix/issue-839-no-clobber-dst-region
Open

fix(cp): apply --destination-region in shouldOverride for --no-clobber (#839)#862
gustcol wants to merge 1 commit intopeak:masterfrom
gustcol:fix/issue-839-no-clobber-dst-region

Conversation

@gustcol
Copy link
Copy Markdown

@gustcol gustcol commented Apr 12, 2026

Context

Fixes #839

Root cause

shouldOverride performs a HEAD request on the destination to determine
whether a copy should be skipped (--no-clobber, --if-size-differ,
--if-source-newer). It created a destination storage.Client using
c.storageOpts directly — but c.storageOpts does not yet have the
destination region applied at that point.

doUpload and doCopy both call c.storageOpts.SetRegion(c.dstRegion)
just before creating their destination clients, but shouldOverride had
no equivalent. When --destination-region points to a non-default region,
the HEAD request was signed for the wrong region, causing:

AWS Error: AuthorizationHeaderMalformed (Status Code: 400)

Fix

Derive a local dstOpts copy inside shouldOverride and apply
dstOpts.SetRegion(c.dstRegion) before constructing the destination
client — matching the pattern already used in doUpload and doCopy.

dstOpts := c.storageOpts
if c.dstRegion != "" {
    dstOpts.SetRegion(c.dstRegion)
}
dstClient, err := storage.NewClient(ctx, dsturl, dstOpts)

Files changed: command/cp.go

Tests

  • Added TestShouldOverride_NoFlagsIsNoop — verifies the fast-path (no override flags) still returns nil, and acts as a compile-time check that dstRegion is correctly wired into the Copy struct and shouldOverride.
  • Full HTTP-level regression requires a multi-region fake S3 and belongs in the e2e suite.

Manual verification

# Before: HTTP 400 when bucket is in eu-central-1
s5cmd cp --no-clobber --destination-region eu-central-1 file.txt s3://eu-bucket/file.txt

# After: HEAD is signed for eu-central-1; --no-clobber skips correctly
s5cmd cp --no-clobber --destination-region eu-central-1 file.txt s3://eu-bucket/file.txt

Fixes peak#839

shouldOverride performs a HEAD request on the destination object to
decide whether to skip the copy.  doUpload and doCopy both called
c.storageOpts.SetRegion(c.dstRegion) before creating the destination
client, but shouldOverride did not.  When --destination-region is set
and the bucket lives in a non-default region, the HEAD was signed for
the wrong region, producing HTTP 400 AuthorizationHeaderMalformed.

Fix: derive a local dstOpts copy inside shouldOverride and set the
region on it before constructing the destination client, matching the
pattern already used in doUpload and doCopy.

Reproducer:
  s5cmd cp --no-clobber --destination-region eu-central-1 \
    local.txt s3://eu-bucket/object.txt
  # Fails: AWS Error: AuthorizationHeaderMalformed (400)

After this change:
  HEAD request is signed for eu-central-1; --no-clobber works correctly.
@gustcol gustcol marked this pull request as ready for review April 12, 2026 15:35
@gustcol gustcol requested a review from a team as a code owner April 12, 2026 15:35
@gustcol gustcol requested review from ilkinulas and seruman and removed request for a team April 12, 2026 15:35
@gustcol
Copy link
Copy Markdown
Author

gustcol commented Apr 12, 2026

Test results

Unit tests (go test -race ./...): PASS

Specific tests:

go test -race -run "TestShouldOverride_NoFlagsIsNoop" ./command/...
ok  github.com/peak/s5cmd/v2/command

Wasabi smoke test:

# Upload file (first time — creates it)
$ ./s5cmd cp -n --destination-region us-east-1 /tmp/file.txt s3://bucket/noclobber-test.txt
cp /tmp/file.txt s3://bucket/noclobber-test.txt

# Second upload with --no-clobber + --destination-region
$ ./s5cmd --log debug cp -n --destination-region us-east-1 /tmp/file.txt s3://bucket/noclobber-test.txt
DEBUG "cp /tmp/file.txt s3://bucket/noclobber-test.txt": object already exists
# Exit: 0 — no AuthorizationHeaderMalformed 400 error

Before this fix, the HEAD request used the wrong region signing, causing HTTP 400. Now shouldOverride applies dstRegion before creating the client.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

shouldOverride (for --no-clobber) should match SetRegion logic of doUpload

1 participant