`nerdctl run` with attached stdio stops draining container stdout at ~72 KiB, deadlocking the container (nerdctl 2.3.5, containerd 2.3.3, Kata runtime-rs)

[gsaddict91]

Summary

With nerdctl run --rm -i (foreground, non-TTY, stdio attached), the attach-side stdout consumer stops reading after roughly 72 KiB have been delivered to the host. The container's writer then blocks forever on a full pipe and the whole chain deadlocks: the container never exits, nerdctl never returns, and signals sent to the nerdctl client do not tear the task down.

The same image, same containerd, same runtime, driven through containerd's own ctr run client instead of nerdctl, streams 64 MiB of stdout cleanly. That is what points at the nerdctl attach path rather than at the runtime or the guest.

Environment

Component	Version / detail
nerdctl	2.3.5 (official release binary, checksum-verified)
containerd	2.3.3
Runtime	Kata Containers 4.0.0, runtime-rs shim (containerd-shim-kata-v2 from the kata-static runtime-rs/bin tree)
Hypervisor actually used	QEMU (qemu-system-x86_64), via the shim's vendored configuration-qemu-runtime-rs.toml
Guest stdio pipe size	Kata container_pipe_size effective value 0 → OS default (~64 KiB)
Host OS	Ubuntu 24.04.4, kernel 6.17.0-1021-azure (x86_64, KVM)
Guest kernel	6.18.35
runc	Not installed on this host — see "What was and was not tested"
BuildKit / CNI	not installed; container had --network none

Reproduction

Everything below is generic; run as root on a host with containerd + nerdctl + a Kata runtime handler registered. $RUNTIME is the Kata shim handler name (ours resolved to the runtime-rs shim); $IMAGE is any image with a POSIX shell and dd already present in the content store (we used --pull never on an air-gapped host).

A. Through nerdctl — stalls

# 4 MiB of stdin, consumed to EOF first, then 8 MiB written to stdout
head -c 4194304 /dev/zero > /tmp/stdin.bin

timeout 90 nerdctl run --rm -i --pull never --runtime "$RUNTIME" "$IMAGE" \
  sh -c 'cat > /dev/null; dd if=/dev/zero bs=1M count=8 2>/dev/null' \
  < /tmp/stdin.bin > /tmp/out.bin
echo "exit=$?"
wc -c < /tmp/out.bin

Observed: nerdctl never returns; timeout fires (exit=124). Sampling /tmp/out.bin while the run is wedged shows delivery starting and then stopping — in our live-sampled reproduction it stopped at exactly 73,728 bytes (72 KiB = a 64 KiB pipe plus an 8 KiB copy buffer) and never advanced again.

B. Same image and runtime through ctr — completes

timeout 90 ctr run --rm --runtime "$RUNTIME" "$IMAGE" probe-ctr \
  sh -c 'dd if=/dev/zero bs=1M count=8 2>/dev/null' > /tmp/out2.bin
echo "exit=$?"
wc -c < /tmp/out2.bin

Observed: exit=0, all 8,388,608 bytes delivered. We ran this control across a size sweep of 0.25 / 0.5 / 1 / 2 / 4 / 8 / 16 / 32 / 64 MiB, with both a single monolithic write() and 64 KiB chunked writes awaiting drain, and with a bidirectional shape (4 MiB in to EOF, then 30 MiB out). Every cell passed, including 64 MiB monolithic (all bytes delivered, ~8.7 s wall including VM boot) and the 30 MiB bidirectional case (~1.2 s in-guest).

Observed vs expected

Expected: nerdctl run with attached stdio keeps draining the container's stdout for as long as the container writes, as ctr does; the container runs to completion and nerdctl exits with the container's status.

Observed:

1. Host-side stdout delivery stops at ~72 KiB (73,728 bytes in the sampled run) and never resumes.
2. The guest writer blocks on the now-full pipe; the container never exits and CPU goes idle.
3. Thread/wchan sampling of the wedged process chain shows the Kata runtime-rs shim's tokio worker blocked in anon_pipe_write — i.e. it is trying to write stdout toward nerdctl — while every nerdctl thread sits idle in futex/epoll. Nothing on the nerdctl side is consuming.
4. Signals do not get through the wedged attach: SIGTERM to the nerdctl client (via timeout) killed nothing; one specimen sat wedged for 19+ minutes.
5. Recovery through nerdctl is also blocked: nerdctl rm --force <name> hangs. Killing the nerdctl client leaves the shim, the hypervisor process and the virtiofsd processes alive (a leaked VM). ctr task kill -s SIGKILL <id> returns 0 but the task stays RUNNING; ctr task rm -f hangs. Only killing the hypervisor and virtiofsd PIDs first, then the shim, reclaims everything — after which ctr container rm and nerdctl rm --force both succeed.

Point 3 is the core of the report: the writer is blocked writing to nerdctl, and nerdctl is asleep. That is a stalled reader on the attach side rather than a runtime or guest-agent problem.

The ~72 KiB threshold

73,728 bytes = 64 KiB + 8 KiB, which lines up with a default-sized (64 KiB) stdio pipe plus one 8 KiB copy buffer already consumed. That is consistent with the attach-side reader performing one buffer's worth of copying and then never scheduling another read, leaving the pipe permanently full. We did not bisect the threshold with a size sweep through nerdctl, so treat 73,728 as one precisely measured wedge point rather than a proven constant; the qualitative finding is that small outputs (diagnostics, short JSON) always got through and every real payload never did.

What was and was not tested

Stated explicitly so nothing here is over-claimed:

• Reproduced: nerdctl 2.3.5 + containerd 2.3.3 + Kata 4.0.0 runtime-rs shim under QEMU, non-TTY, --interactive, with stdin fed and read to EOF before the large stdout write. Reproduced with a real workload emitting a large JSON result, and confirmed by live thread/wchan sampling of the wedged chain.
• Ruled out (same host, same runtime, same image): the Kata guest, the guest agent, and the runtime-rs shim's own stdio copy path — ctr drains the identical FIFOs at up to 64 MiB without issue, monolithic and chunked alike. Request/stdin size was never a factor (a 35 KB stdin request reproduced the stall).
• NOT tested: runc. runc is deliberately not installed on this host (the only registered runtime is the Kata shim), so we have no data on whether the same stall occurs with nerdctl run --runtime runc. We therefore cannot say whether this is Kata-specific or a general property of nerdctl's attach path. The evidence isolates the fault to nerdctl's attach-side reader relative to ctr on the same runtime, and no further.
• NOT tested: TTY mode (-t), detached mode with nerdctl logs, running without --interactive/without stdin attached, other log drivers, and other nerdctl versions. Every nerdctl reproduction we have involves -i with stdin attached, so we cannot say whether stdin attachment is required to trigger it.
• NOT tested: whether raising Kata's container_pipe_size (which is honored end-to-end down to F_SETPIPE_SZ in the guest, but is 0/default in the shipped QEMU runtime-rs template) merely raises the threshold. It would not address a reader that stops reading.

Impact / workaround

Any workload whose stdout exceeds ~72 KiB under nerdctl run with attached stdio deadlocks permanently and leaks the sandbox VM, since neither signal proxying nor nerdctl rm --force can reach it. Our workaround was to move all bulk output off stdio entirely (onto a bind-mounted output file), keeping stdio for small control-plane messages only, plus an out-of-band PID-kill escalation for teardown. Happy to run further probes on this host if a maintainer wants a specific variant tested (TTY, no-stdin, alternate log driver).

[ogulcanaydogan]

Picking this up. Really thorough report, appreciate the wchan sampling and the runc control you tried to run (even though runc wasn't available on that host). Going to try to reproduce with runc on a plain Linux box first, to narrow down whether this is nerdctl's attach-side reader in general or something specific to the Kata runtime-rs shim's stdio handling.

[ogulcanaydogan]

Reproduced your control methodology with runc on a plain Linux box (nerdctl 2.3.5, containerd 2.3.3, runc 1.5.1, same versions as your report), since Kata wasn't an option here (no nested virtualization on this host either).

Ran your exact repro (4 MiB stdin to EOF, then stdout write, -i, non-TTY) and extended it into a size sweep plus a chunked-write variant:

OK   size=1MiB   exit=0
OK   size=4MiB   exit=0
OK   size=8MiB   exit=0
OK   size=16MiB  exit=0
OK   size=32MiB  exit=0
OK   size=64MiB  exit=0
chunked 64KiB writes, 8MiB total: exit=0, all bytes delivered

Every case delivered the full byte count with exit 0, no stall at 72 KiB or anywhere else. This lines up with your own runc control from other angles (monolithic and chunked writes both fine), and points away from a general bug in nerdctl's attach-side reader.

Given that, this looks like it's specific to the Kata runtime-rs shim's stdio path (or something in how it interacts with nerdctl's cioutil/cio wiring specifically), not nerdctl's attach handling in general. I don't have a Kata environment to take it further from here. Might be worth also filing this against kata-containers/kata-containers if you haven't already, since the evidence now points there rather than at nerdctl's own code.

[gsaddict91]

Thanks for running the runc control. Since then I've traced the relevant code (nerdctl v2.3.5, containerd v2.3.3, kata-containers 4.0.0 runtime-rs) and I can now reproduce the deadlock under runc, with no Kata involved, at the same byte offset I reported. So the bug is in nerdctl's attach path after all, but it needs a second, internal process to misbehave before it shows up, which is why the plain runc control passes.

What actually happens

nerdctl run in the foreground doesn't write container stdout straight to the terminal. It always sets up the json-file log driver, detached or not (pkg/cmd/container/create.go:274-288, "Always set internalLabels.logURI"). At IO setup time it forks a logger child (nerdctl _NERDCTL_INTERNAL_LOGGING ...) and tees output through io.MultiWriter(os.Stdout, loggerPipe) on the same goroutine that reads the stdout FIFO (pkg/cioutil/container_io.go:153-201). The logger pipe is a 64 KiB anonymous pipe.

If the logger child ever stops consuming, for any reason, the tee write blocks forever:

• nerdctl keeps its own copies of the logger pipe's read ends open. The closers for them only run on the error path (container_io.go:128-137,159,165); the CloseAfterStart step that containerd's version of this code has (cmd/containerd-shim-runc-v2/process/io.go:331-343) is missing. Because nerdctl itself still holds a read end, the kernel never raises EPIPE and the write just parks.
• The parked goroutine is the FIFO reader, so the stdout FIFO fills up and the runtime's writer blocks behind it. From the outside this looks exactly like my wchan sample: the shim blocked in anon_pipe_write, every nerdctl thread idle in futex/epoll (a Go pipe write parked on the netpoller).
• The stall offset is 64 KiB (pipe capacity) plus one writer chunk. The kata runtime-rs shim writes stdout in 8 KiB chunks (tokio::io::copy's buffer size; QEMU has no passfd io so it's on the legacy ShimIo path), and 65536 + 8192 = 73728, the number I measured.

Reproduction without Kata

Plain Linux (privileged container), containerd 2.3.3 + nerdctl 2.3.5 + runc 1.5.1, busybox:

nerdctl run --rm --net none --name wedge busybox sh -c \
  'sleep 3; i=0; while [ $i -lt 1024 ]; do dd if=/dev/zero bs=8192 count=1 2>/dev/null; sleep 0.02; i=$((i+1)); done' \
  > /tmp/out.bin &
sleep 1.5
# simulate the logger stopping (SIGTERM and SIGKILL behave the same here)
kill -TERM "$(pgrep -f _NERDCTL_INTERNAL_LOGGING | head -1)"
# then sample: wc -c /tmp/out.bin

Results, all deterministic across repeats:

• Logger untouched (control): 8 MiB streamed, exit 0.
• Logger killed (TERM or KILL): out.bin freezes at 73728 exactly and never advances; nerdctl never exits.
• Same but without -i/stdin: identical wedge. Stdin attachment is irrelevant; the -i correlation in my report was a coincidence.
• --log-driver none: streams fine. The tee is the wedge.
• Writer using 32 KiB chunks instead of 8 KiB: freezes at 98304 = 64Ki + 32Ki. The offset really is "pipe plus one chunk", which ties my 73728 to the kata shim's 8 KiB copy buffer.

Forensics while wedged: /proc/<nerdctl>/fd shows the same pipe inode twice per stream (the leaked read ends); the container's json log is 0 bytes; the killed logger stays around as a zombie (nerdctl only reaps it in ncio.Close). Teardown also misbehaves under runc: nerdctl rm -f wedge hung past a 10 s timeout, and even after force-removing the task with ctr, the nerdctl client stays stuck forever, since its stdout copier can never return.

Why the earlier controls passed

• Your runc sweep (and mine) passes because a healthy logger keeps draining the pipe; nothing on the happy path blocks.
• ctr passes under the same kata shim because it has no tee; it writes to os.Stdout directly (cmd/ctr/commands/tasks/tasks_unix.go:96-112, pkg/cio/io_unix.go:56-107).
• My kata runs hit it because something on that path stops the logger. That initiator is the one piece I haven't pinned down in source. One candidate: the logger cancels its readers on anything arriving from its task-wait channel (pkg/logging/logging.go:376-381 is a bare <-exitCh), and containerd's client puts Wait RPC errors into that channel as a synthetic exit (client/task.go:342-368). So a transient Wait failure against the kata shim would silently kill logging and, via the leaked read ends, wedge the attach. I'll run the discriminating checks on the kata host (is the logger process a zombie or alive-but-not-draining, json-log size at wedge time, strace on the logger around the stall) and report back.

Suggested fixes

1. Close the parent's copies of the logger pipe read ends after cmd.Start() (the missing CloseAfterStart equivalent). Logger death then surfaces as EPIPE instead of a permanent deadlock that also takes rm -f down with it.
2. In the logger, check ExitStatus.Error() before treating a Wait-channel delivery as a container exit (logging.go:376-381).
3. Maybe worth discussing separately: foreground attach output probably shouldn't be gated on a blocking same-goroutine pipe write to a best-effort logging process at all.
4. The 1-byte ready handshake treats EOF as success (container_io.go:192-195), unlike containerd's n == 0 check (process/io.go:315-317), so a logger that dies before ready() still gets wired into the tee.

Workaround

--log-driver none removes the tee entirely. Much cleaner than moving bulk output off stdio, which is what I had been doing.

The kata side

The kata runtime-rs shim doesn't cause the stall, but it's what makes it unrecoverable: it holds its own read end of the stdout FIFO (so a stalled or departed client reader can never produce EPIPE), and it gates all exit reporting on the stdio copy tasks finishing, which is why kill reported success while the task stayed RUNNING, rm -f hung, and the sandbox VM leaked. That part is independent of nerdctl, so I've filed it separately with source references: kata-containers/kata-containers#13676.

[ogulcanaydogan]

Independently reproduced this without Kata, following your steps: started a container writing 8 KiB chunks with a small sleep between them, killed the internal _NERDCTL_INTERNAL_LOGGING process mid-stream, and the output permanently froze (5,783,552 bytes, a clean multiple of the chunk size). nerdctl run never returned, and nerdctl rm -f on the wedged container hung as well, matching what you described. Real, deterministic bug, not a fluke or a Kata artifact, exactly as you diagnosed.

Great tracing work, genuinely, that's a hard bug to pin down. Are you planning to submit a PR for the fixes you outlined (the missing CloseAfterStart-equivalent, the Wait-channel error check, the ready-handshake EOF check)? Don't want to step on it a second time this week. Happy to help review/test if useful, or leave it to you if you've already got it in hand.

[gsaddict91]

PR is up: #5151.

It ended up needing one addition beyond the three fixes I listed. Closing the parent's read ends alone is necessary but not sufficient: with that in place the logger's death does surface as EPIPE, but the error kills the io.MultiWriter, which aborts the io.CopyBuffer draining the stdout FIFO — and the container still wedges behind the undrained FIFO chain (verified with /proc fd forensics on a wedged specimen: read ends gone, stdout copy goroutine exited, run still hung). So the "attach shouldn't be gated on a best-effort logging process" point turned out to be required, not optional: the logger leg of the tee is now best-effort — it warns once and drops out on the first failed write while the attach keeps streaming.

With both halves in, your repro (and mine, TERM and KILL variants) delivers all 8 MiB with exit 0 and clean teardown even when the logger is killed mid-stream; details and an A/B table are in the PR.

Review and a test run on your repro setup would be very welcome.