docker buildx build --push fails with failed to fetch oauth token after long-running builds against Google Artifact Registry using gcloud credential helper

[roberto-rodriguez-paack]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

I'm seeing what appears to be an authentication issue during the registry export phase of BuildKit.

A long-running docker buildx build --push fails when pushing to Google Artifact Registry, while the exact same image can be pushed successfully afterwards using docker push. Same failure happens when the flag 'cache-to' is pointing to Google Artifact Registry

The failure only happens after builds that run for several minutes.

Environment

• Docker Engine: 27.0.3
• Docker Buildx: v0.35.0
• BuildKit: 0.31.2
• Buildx driver: docker-container

Authentication

Authentication is configured using Google's Docker credential helper:

gcloud auth configure-docker europe-west1-docker.pkg.dev

Reproduction

Run a build that takes several minutes:

Expected behaviour

docker buildx build --push should behave equivalently to docker buildx build --load followed by docker push, since both use the same Docker credential helper and authentication remains valid after the build completes.

Actual behaviour

ERROR: failed to push: failed to authorize: failed to fetch oauth token: unauthorized: authentication failed

Buildx version

github.com/docker/buildx v0.35.0 a319e5b

Docker info

Client:
 Version:    27.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.35.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 29.6.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 11ce9d5f3c68c941867e82890e93e815c1304f1b
 runc version: v1.3.6-0-g491b69b
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.154+
 Operating System: Alpine Linux v3.24 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.63GiB
 Name: cd11ef144c6f
 ID: 604eaf98-41f0-4919-85f2-35cfc1438832
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
[DEPRECATION NOTICE]: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
In future versions this will be a hard failure preventing the daemon from starting! Learn more at: https://docs.docker.com/go/api-security/
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Builders list

NAME/NODE                           DRIVER/ENDPOINT                  STATUS     BUILDKIT   PLATFORMS
buildkit-builder-15561755384*       docker-container                                       
 \_ buildkit-builder-155617553840    \_ tcp://docker:2375            inactive              
default                             docker                                                 
 \_ default                          \_ default                      running    v0.31.2    linux/amd64 (+4), linux/arm64, linux/arm (+2), linux/ppc64le, (4 more)
docker-context-15561755384          docker                                                 
 \_ docker-context-15561755384       \_ docker-context-15561755384   running    v0.31.2    linux/amd64 (+4), linux/arm64, linux/arm (+2), linux/ppc64le, (4 more)

Configuration

FROM alpine

RUN sleep 360

Then:

docker buildx build \
  --push \
  -t europe-west1-docker.pkg.dev/.../image:latest \
  .

Build logs

Additional info

Observed behavior

The build completes successfully.

The registry export fails with:

failed to push europe-west1-docker.pkg.dev/...: failed to authorize: failed to fetch oauth token: unauthorized: authentication failed

Additional observations

Immediately after the failure:

gcloud auth print-access-token

still succeeds.

Likewise:

docker push europe-west1-docker.pkg.dev/.../image:latest

using the exact same credentials also succeeds.

Therefore:

• Google authentication is still valid.
• The Docker credential helper is still able to obtain fresh access tokens.
• Only the registry export performed by docker buildx build --push fails.

Experiments performed

Test	Result
Build <2 minutes	✅ Success
Build >5 minutes	❌ failed to fetch oauth token
gcloud auth print-access-token before build	✅
gcloud auth print-access-token after failure	✅
docker buildx build --load + docker push	✅
BUILDKIT_NO_CLIENT_TOKEN=1	Changes the failure to 405 Method Not Allowed