panic: send on closed channel in Printer.Write via LoadImage (--load) progress after solve completes

[fedor-rubalskiy]

Summary

docker buildx build --load can crash the client with panic: send on closed channel in util/progress.(*Printer).Write when the local image load (LoadImage / "importing to docker") is still streaming progress after the build's top-level progress Printer has been closed. It is non-deterministic and only shows up on large images whose local load is slow enough to outlive the solve.

This is the same underlying defect as #3875, but on a different writer path. #3875 was closed by #3886, which reordered cleanup so the remote policy-resolver progress writer no longer outlives the printer. The LoadImage / --load writer (util/dockerutil) is not covered by that fix, and Printer.Write itself is still an unguarded channel send on master.

Version

Observed on a self-hosted CI runner using Docker's docker-buildx plugin. The racing code is unchanged on the latest release v0.35.0 and current master (verified by source inspection — see below), so this is not specific to an old version.

Stack trace

panic: send on closed channel

goroutine 1637348 [running]:
github.com/docker/buildx/util/progress.(*Printer).Write(...)
	github.com/docker/buildx/util/progress/printer.go:89 +0x26
github.com/docker/buildx/util/progress.(*pw).Write(...)
	github.com/docker/buildx/util/progress/reset.go:66 +0xa5
github.com/docker/buildx/util/progress.Wrap.func1()
	github.com/docker/buildx/util/progress/progress.go:36 +0x1be
panic({...})
	runtime/panic.go:860 +0x13a
github.com/docker/buildx/util/progress.(*Printer).Write(...)
	github.com/docker/buildx/util/progress/printer.go:89 +0x26
github.com/docker/buildx/util/progress.(*pw).Write(...)
	github.com/docker/buildx/util/progress/reset.go:66 +0xa5
github.com/docker/buildx/util/progress.(*subLogger).SetStatus(...)
	github.com/docker/buildx/util/progress/progress.go:95 +0xc2
github.com/docker/buildx/util/dockerutil.fromReader.func1()
	github.com/docker/buildx/util/dockerutil/progress.go:23 +0x1de
github.com/docker/buildx/util/dockerutil.fromReader(...)
	github.com/docker/buildx/util/dockerutil/progress.go:39 +0x7aa
github.com/docker/buildx/util/dockerutil.(*Client).LoadImage.func1.2(...)
	github.com/docker/buildx/util/dockerutil/client.go:64 +0x45
github.com/docker/buildx/util/progress.Wrap(...)
	github.com/docker/buildx/util/progress/progress.go:47 +0x297
github.com/docker/buildx/util/dockerutil.(*Client).LoadImage.func1()
	github.com/docker/buildx/util/dockerutil/client.go:63 +0x245
created by github.com/docker/buildx/util/dockerutil.(*waitingWriter).Write.func1 in goroutine 1637362
	github.com/docker/buildx/util/dockerutil/client.go:111 +0x1b

The log immediately before the panic shows the image load in progress (two multi-GB layers), which is what gives the race a wide enough window:

#57 loading layer dbd9cc758bf1 2.68GB / 2.70GB 33.0s
#57 loading layer 96431b6a72fe 777.09MB / 2.98GB 10.2s
panic: send on closed channel

Root cause

Printer.Write sends on p.status with no guard (master, util/progress/printer.go):

func (p *Printer) Write(s *client.SolveStatus) {
	p.status <- s
	...
}

func (p *Printer) Wait() error {
	p.closeOnce.Do(func() {
		close(p.status)
	})
	<-p.done
	return p.err
}

closeOnce only makes the close idempotent; it does nothing to prevent a Write that races the close.

On --load, LoadImage streams "importing to docker" progress into that same Printer from a goroutine (util/dockerutil/client.go):

func (w *waitingWriter) Write(dt []byte) (int, error) {
	w.once.Do(func() {
		go w.f() // f() runs progress.Wrap("importing to docker", status.Write, ...) -> Printer.Write
	})
	return w.PipeWriter.Write(dt)
}

waitingWriter.Close() waits on its own w.done, so the load pipe is synchronized — but the goroutine writes into the build's top-level Printer, whose Wait() is called elsewhere in the build pipeline. When the local load is slow (large image), that Wait()/close(p.status) can happen while the "importing to docker" goroutine is still calling Printer.Write, and the send lands on a closed channel → panic.

This is exactly the "progress writer outlives the printer" class that #3886 fixed for the policy-resolver writer; the --load/LoadImage writer needs the same guarantee (or Printer.Write needs to stop panicking on a post-close write).

Suggested fix

Either (or both):

1. Make Printer.Write non-fatal after close, e.g.

   func (p *Printer) Write(s *client.SolveStatus) {
       select {
       case p.status <- s:
       case <-p.done:
           return // printer already closed; drop late status
       }
       ...
   }

   This makes any late writer (LoadImage, and future callers) safe rather than fixing one caller at a time.

2. Ensure the --load "importing to docker" goroutine is joined before the build's Printer.Wait() closes the channel, mirroring the cleanup reordering in build: fix policy resolver cleanup on errors #3886.

Impact

Non-deterministic red builds on --load of large images (multi-GB), even though the image built and every layer exported — the crash is purely in the client's progress plumbing. Happy to test a patch.

[tonistiigi]

The only way I can see this being possible is with build cancellation, but your report does not mention it.

There is a w.done wait in the loader WriteCloser close method that should block before printer.Wait() is allowed to be called.