Replies: 23 comments 27 replies
|
Hello @Anamika1608, please take a look on the above plan and let us know if everything is ok. |
|
hi, yeah. plan looks good to me. |
|
Hi @cmainas my only machine right now is an Apple Silicon Mac (M4). apple's hypervisor doesn't expose nested virtualization, so i can't get /dev/kvm inside a local Linux VM, which rules out Firecracker and Cloud Hypervisor locally, since both need KVM to start. QEMU still runs under TCG without KVM, and its QMP control socket works in that mode too. could you confirm a few things:
thanks! |
|
Hi @Anamika1608! Regarding your first point, according to this https://u3n.medium.com/the-future-of-development-is-here-running-firecracker-microvms-on-your-macbook-pro-m3-ad6fd3e5092c you can get FC working on an apple Lima VM. Would it worth a try? Regardless, we can provide a host for your dev/tests. Cheers, |
|
Hello @Anamika1608 , for the third question,, we should be able to somehow access the socket (and therefore interact with the guest). It does not need to be from the host namespace, it is totally fine if we should enter the urunc's container namespace. |
|
@Anamika1608 the link to the kata-containers runtime-rs FC refactor I mentioned: https://github.com/kata-containers/kata-containers/tree/fix/runtime-rs-net-fc |
|
i have a working socket PoC. in urunc, firecracker.go now launches firecracker --api-sock instead of --no-api --config-file, and i added an HTTP-over-Unix client (firecrackerClient) that does waitForSocket -> PUT /machine-config, /boot-source, /drives/ -> PUT /actions {InstanceStart}. the socket is a per-container pathname socket. an integration test drives a real Firecracker over the api-sock with this client, and a real guest boots, the serial shows InstanceStart returning 204 and the kernel coming up (Linux version 6.1.174). i'm running it on an Apple-Silicon Lima VM with nested virtualization, so it's real KVM. there are also unit tests against a fake API server that need no KVM. here is the committed code - Anamika1608@d601cbe also here is the doc - https://docs.google.com/document/d/1lPLB9skvf7s7l8uB9Nz-YK7Zvy0W9NYe61dSxjkVe2w/edit?usp=sharing two questions:
|
|
Hello @Anamika1608 , that is great! Regarding your questions:
|
|
hi @cmainas, i checked the published Firecracker images ( |
|
Hello @Anamika1608 thank you for checking them out. We need to provide multi-arch images. But did you also try |
|
@cmainas, it worked. |
|
so, i have successfully connected to vmm via socket from outside of urunc, and ran the urunc as the containerd. it passed. i did sent the vmm info curn request and it did succeed. 
i have also added my whole POC process in this doc with the theory that we needed for phase 1 - https://docs.google.com/document/d/1lPLB9skvf7s7l8uB9Nz-YK7Zvy0W9NYe61dSxjkVe2w/edit?usp=sharing check once? and let me know if this completes our phase 1. thanks! |
|
Hello @Anamika1608 , thank you for the report. Some generic comments:
Regarding the report, the text looks good (just some corrections regarding urunc) and it is enough. A couple questions:
and some notes:
For the PoC is the code in this branch main...Anamika1608:urunc:poc/fc-api-sock ? |
|
hi @cmainas, i have done all the mentioned changes, doc is detailed. check once. regarding your questions
|
PHASE 1: THEORETICAL BACKGROUND + POCScope of this sectionPhase 1 covers the fundamental theory: Linux namespaces, IPC via Unix sockets, and how urunc manages these today. I analyzed the urunc source and verified the behavior of control sockets through manual testing. Every detail here is backed by the Linux man pages, urunc codebase, or official documentation. Namespaces and socket placementurunc isolates unikernels by placing the VMM inside specific Linux namespaces. Because a control socket is needed for runtime configuration, namespace boundaries dictate where this socket must live. If the runtime cannot cross the namespace wall, it cannot talk to the VMM. This makes namespace theory the primary constraint for our design. Namespace OverviewNamespaces isolate global system resources so they appear private to a process group. Per
The API uses three main syscalls: For this project, the network namespace (where the VMM sits) and the mount namespace (which controls file visibility) are critical. Note: the IPC namespace does not isolate Unix domain sockets. Unix Socket Address TypesUnix domain sockets provide local IPC using three address formats defined in
Pathname sockets require specific filesystem permissions, while abstract sockets bypass standard file permissions entirely. Impact of Namespaces on IPCTwo key facts determine how we communicate across namespaces: Fact 1: Abstract sockets are trapped. Fact 2: Pathname sockets are filesystem-bound. Visibility depends on the mount namespace, not the network namespace. If the host can see the file, it can connect (pending permissions). I verified this with a demo: an abstract socket listener in a netns was invisible to the host (Connection refused), but a pathname socket in Socket Location ConstraintsTo allow the host-side runtime to talk to the jailed VMM:
Since urunc can enter the container's namespace using Current urunc Namespace UsageThe existing urunc flow already handles complex isolation:
urunc Internal IPCurunc already utilizes pathname sockets for its The VMM Boot GapCurrently, urunc disables control channels before Industry Comparisons
Impact on PoC DesignThe theory led to a single design choice: a pathname socket per container. The PoC swaps References
SETTING UP THE ENVIRONMENT FOR FIRECRACKER VMM AND IF KVM EXISTS1) create & start the nested-virt VMlimactl start --name=fc-vm --vm-type=vz --set='.nestedVirtualization=true' template://defaultLima prints the VM config (CPUs/memory/disk) and asks you to confirm, choose "Proceed with the current configuration." Flags recap: It's done when you get your shell prompt back with a line like 2) is /dev/kvm there?limactl shell fc-vm -- sh -c 'uname -m; ls -l /dev/kvm'Success = an architecture line (aarch64) followed by a device line like INSTALLING FIRECRACKER1) step into your Linux boxlimactl shell fc-vm2) install firecrackercd ~
ARCH="$(uname -m)"
VERSION="v1.7.0"
curl -L https://github.com/firecracker-microvm/firecracker/releases/download/${VERSION}/firecracker-${VERSION}-${ARCH}.tgz | tar -xz
sudo mv release-${VERSION}-${ARCH}/firecracker-${VERSION}-${ARCH} /usr/local/bin/firecracker
rm -rf release-${VERSION}-${ARCH}
firecracker --versionFirecracker v1.7.0 installed and it ran cleanly. DOWNLOAD GUEST KERNEL + DISK IMAGERun these inside the VM. First a working directory and the prep tools: sudo apt-get update && sudo apt-get install -y squashfs-tools jq e2fsprogs
mkdir -p ~/fc && cd ~/fcDownload the guest kernel (vmlinux), it auto-discovers the latest aarch64 CI kernel: ARCH="$(uname -m)"
S3="https://s3.amazonaws.com/spec.ccfc.min"
CI_ARTIFACTS_PREFIX=$(curl -fsSL "$S3?list-type=2&prefix=firecracker-ci/&delimiter=/" \
| grep -oP "(?<=<Prefix>)firecracker-ci/[0-9]{8}-[^/]+/(?=</Prefix>)" \
| sort | tail -1)
latest_kernel_key=$(curl -fsSL "$S3?list-type=2&prefix=${CI_ARTIFACTS_PREFIX}${ARCH}/vmlinux-" \
| grep -oP "(?<=<Key>)(${CI_ARTIFACTS_PREFIX}${ARCH}/vmlinux-[0-9]+\.[0-9]+\.[0-9]{1,3})(?=</Key>)" \
| sort -V | tail -1)
wget "$S3/${latest_kernel_key}"Download + build the root filesystem (Ubuntu, converted to a writable ext4 with an SSH key baked in for later login): latest_ubuntu_key=$(curl -fsSL "$S3?list-type=2&prefix=${CI_ARTIFACTS_PREFIX}${ARCH}/ubuntu-" \
| grep -oP "(?<=<Key>)(${CI_ARTIFACTS_PREFIX}${ARCH}/ubuntu-[0-9]+\.[0-9]+\.squashfs)(?=</Key>)" \
| sort -V | tail -1)
ubuntu_version=$(basename $latest_ubuntu_key .squashfs | grep -oE '[0-9]+\.[0-9]+')
wget -O ubuntu-$ubuntu_version.squashfs.upstream "$S3/$latest_ubuntu_key"
unsquashfs ubuntu-$ubuntu_version.squashfs.upstream
ssh-keygen -f id_rsa -N ""
cp -v id_rsa.pub squashfs-root/root/.ssh/authorized_keys
mv -v id_rsa ./ubuntu-$ubuntu_version.id_rsa
sudo chown -R root:root squashfs-root
truncate -s 1G ubuntu-$ubuntu_version.ext4
sudo mkfs.ext4 -d squashfs-root -F ubuntu-$ubuntu_version.ext4then run below inside fc directory ls -lh ~/fcCONNECTING WITH FIRECRACKER VMM VIA SOCKETTerminal 1 — start Firecrackerwe already have one shell in the VM. In it: cd ~/fc
API_SOCKET="/tmp/firecracker.socket"
sudo rm -f $API_SOCKET
sudo /usr/local/bin/firecracker --api-sock "${API_SOCKET}"It prints a couple of lines and then sits there doing nothing. That's exactly right, it just built an empty VM and is waiting for instructions on the socket. Leave it running. (We use sudo so it can open Terminal 2 — open a second shell into the VM and drive itOn your Mac, open a new terminal tab and run: limactl shell fc-vm
cd ~/fcThen paste this whole block, each curl configures the empty VM over the socket: API_SOCKET="/tmp/firecracker.socket"
# 1) tell it which KERNEL to boot + kernel command line
KERNEL="./$(ls vmlinux* | tail -1)"
sudo curl -X PUT --unix-socket "${API_SOCKET}" \
--data "{\"kernel_image_path\": \"${KERNEL}\", \"boot_args\": \"console=ttyS0 reboot=k panic=1 pci=off\"}" \
"http://localhost/boot-source"
# 2) give it the DISK (rootfs)
ROOTFS="./$(ls *.ext4 | tail -1)"
sudo curl -X PUT --unix-socket "${API_SOCKET}" \
--data "{\"drive_id\": \"rootfs\", \"path_on_host\": \"${ROOTFS}\", \"is_root_device\": true, \"is_read_only\": false}" \
"http://localhost/drives/rootfs"
# 3) CPUs + memory
sudo curl -X PUT --unix-socket "${API_SOCKET}" \
--data "{\"vcpu_count\": 2, \"mem_size_mib\": 1024}" \
"http://localhost/machine-config"
# 4) POWER ON
sudo curl -X PUT --unix-socket "${API_SOCKET}" \
--data "{\"action_type\": \"InstanceStart\"}" \
"http://localhost/actions"IMPLEMENTATION OF ABOVE
ABSTRACT AND PATHNAME SOCKETSetupsudo apt-get install -y socat
sudo ip netns add demo # create a new netns called "demo"
ip netns list # confirm it existsDemo A: an ABSTRACT socket is trapped in the netns# Listener bound to an ABSTRACT socket, INSIDE the demo netns (background):
sudo ip netns exec demo socat ABSTRACT-LISTEN:mysock,fork SYSTEM:'echo HELLO-FROM-NETNS' &
sleep 1
echo "=== try to reach it from the HOST netns (expect FAIL) ==="
socat -T2 - ABSTRACT-CONNECT:mysock ; echo "exit code = $?"
echo "=== try to reach it from INSIDE the demo netns (expect HELLO) ==="
sudo ip netns exec demo socat -T2 - ABSTRACT-CONNECT:mysock ; echo "exit code = $?"From the host: a "Connection refused" error, exit code = 1 -> The abstract socket simply doesn't exist in the host's network namespace. Demo B: a PATHNAME socket crosses the netns wallsudo pkill -f ABSTRACT-LISTEN # stop the previous listener
sudo rm -f /tmp/path.sock
# Listener bound to a PATHNAME socket (a file in /tmp), still INSIDE demo netns:
sudo ip netns exec demo socat UNIX-LISTEN:/tmp/path.sock,fork SYSTEM:'echo HELLO-VIA-PATH' &
sleep 1
echo "=== the socket FILE is visible from the host, even though the listener is in demo ==="
ls -l /tmp/path.sock
echo "=== try to reach it from the HOST netns (expect HELLO this time) ==="
socat -T2 - UNIX-CONNECT:/tmp/path.sock ; echo "exit code = $?"The pathname socket reaches across the network namespace. That file is visible from the host, even though the listener is inside demo. But the connection was blocked by permissions, not by the namespace, because "connecting to a stream socket object requires write permission on that socket." (I ran the client as user anamika, no sudo) One-time setup – make the Mac <-> VM share writable# 1) stop the VM
limactl stop fc-vm
# 2) make the home share writable + fast (virtiofs suits your vz VM)
limactl edit fc-vm -y \
--set '.mountType = "virtiofs"' \
--set '.mounts[0].writable = true'
# 3) start it back up
limactl start fc-vm
# 4) verify it's now writable from inside the VM
limactl shell fc-vm -- sh -c 'touch /Users/anamika/open-source/urunc-dev/urunc/.wtest && echo WRITABLE && rm -f /Users/anamika/open-source/urunc-dev/urunc/.wtest'Step 4 should print WRITABLE. If it does not print WRITABLE then it is possible the response is stale, run this: limactl shell fc-vm -- sudo mount -o remount,rw /Users/anamika
limactl shell fc-vm -- sh -c 'findmnt -no OPTIONS /Users/anamika; touch /Users/anamika/open-source/urunc-dev/urunc/.wtest 2>&1 && echo WRITABLE && rm -f /Users/anamika/open-source/urunc-dev/urunc/.wtest || echo STILL-READONLY'TO RUN THE URUNC CODE IN VMStep A – make your branch (on the Mac)cd /Users/anamika/open-source/urunc-dev/urunc
git checkout -b poc/fc-api-sock
git branch # confirm: * poc/fc-api-sockStep B – install Go + build deps (inside the VM)limactl shell fc-vmThen, inside the VM: # build prerequisites (gcc for cgo, libseccomp for urunc's seccomp bits)
sudo apt-get update && sudo apt-get install -y build-essential pkg-config libseccomp-dev git
# Go 1.25.4 (urunc's pinned version), arm64
curl -LO https://go.dev/dl/go1.25.4.linux-arm64.tar.gz
sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.25.4.linux-arm64.tar.gz
rm go1.25.4.linux-arm64.tar.gz
# put Go on your PATH
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc && source ~/.bashrc
go versionExpect: Step C – baseline build of the unchanged branch (inside the VM)This compiles urunc as-is, to prove Go + deps + the writable mount all work before you change anything: cd /Users/anamika/open-source/urunc-dev/urunc
go build ./...NOW MAKE THE FIRST COMMIT CHANGES LOCALLY - poc/fc-api-sockRUNNING THE INTEGRATION TESTin the vm: cd /Users/anamika/open-source/urunc-dev/urunc
sudo env \
FC_KERNEL="$(ls $HOME/fc/vmlinux* | tail -1)" \
FC_ROOTFS="$(ls $HOME/fc/*.ext4 | tail -1)" \
/usr/local/go/bin/go test -tags integration -v \
-run TestFirecrackerSocketBoot_Integration ./pkg/unikontainers/hypervisors/Firecracker now launches with a control socket ( TESTING THE REAL URUNC TO TALK TO VMM VIA SOCKETRUN THE CONTAINERDRun in the VM: # 1) create a default containerd config
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
# 2) install + start the containerd systemd service (matching v2.3.2)
sudo wget -qO /etc/systemd/system/containerd.service \
https://raw.githubusercontent.com/containerd/containerd/v2.3.2/containerd.service
sudo systemctl daemon-reload
sudo systemctl enable --now containerd
# 3) verify
echo "active: $(systemctl is-active containerd)"
sudo ctr version | grep -A1 Server
sudo ctr plugin ls | grep "snapshotter.v1" | awk '{print $2, $NF}'Build + install urunc from your branchThis builds urunc + containerd-shim-urunc-v2 from your poc/fc-api-sock branch (so the --api-sock change is baked in) and installs them. In the VM: cd /Users/anamika/open-source/urunc-dev/urunc
# confirm you're on your PoC branch
git branch --show-current
# build, then install the binaries to /usr/local/bin
make
sudo make install
# verify
which urunc containerd-shim-urunc-v2
urunc --version 2>&1 | head -3Register urunc as a containerd runtimeThis adds a small runtime block to containerd's config (using overlayfs, since we skip devmapper), then restarts. In the VM: sudo tee -a /etc/containerd/config.toml >/dev/null <<'EOF'
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.urunc]
runtime_type = "io.containerd.urunc.v2"
container_annotations = ["com.urunc.unikernel.*"]
pod_annotations = ["com.urunc.unikernel.*"]
snapshotter = "overlayfs"
EOF
sudo systemctl restart containerd
echo "active: $(systemctl is-active containerd)"Run a Firecracker container, no guest boot, then reach the socketIMG=harbor.nbfc.io/nubificus/urunc/chttp-firecracker-linux-aarch64:latest
sudo nerdctl rm -f fc-poc 2>/dev/null # clear any earlier attempt
sudo nerdctl pull "$IMG"
sudo nerdctl run -d --name fc-poc --runtime io.containerd.urunc.v2 "$IMG"; echo "run exit: $?"
sleep 2
echo "=== containers ==="
sudo nerdctl ps -a
echo "=== firecracker process (look for --api-sock) ==="
pgrep -af firecracker || echo "no firecracker process"5) Connect to that socket from the host (outside urunc) and query the VMMRun this in the VM: NOTE - the FCPID and NAME is from this demo example run, to get your run FCPID and NAME, run this: FCPID=30868
NAME=f89234e874aacddf21efb5e5a78c0b63ecfead0fc579d16e78c1b6ffe5e90741.sock
echo "=== the VMM socket, seen from the host (outside urunc) ==="
sudo ls -l /proc/$FCPID/root/tmp/$NAME
echo "=== connect to the socket and ask the VMM about itself ==="
sudo curl -s --unix-socket /proc/$FCPID/root/tmp/$NAME http://localhost/ ; echoWhat you should get back is Firecracker's instance info, something like: {"id":"anonymous-instance","state":"Not started","vmm_version":"1.7.0","app_name":"Firecracker"}DIAGRAM SHOWING CONNECTION
|
|
Hello @Anamika1608 , thank you for the report and the PR. Just to make communication easier, let;s name the two approaches we have as:
A few notes from the sync about the next steps:
Please let us know if you need any help with the parts in |
|
following up on the sync notes. here is what i tried, what i committed, what i found, and the questions i have. 1. socket path is now configurableadded a 2. added a boot_mode field so both approaches are real, selectable optionsto actually test and compare both approaches inside real urunc (not a shell script or a go test harness), i added a
both are real, working, testable paths in urunc now, not just flags i experimented with by hand. 3. what i tried to parallelize for api-based, and whyapi-based on its own costs a bit more than config-file-based (a few sequential http calls over the socket vs one config file read), so the only way it wins overall is if that cost gets hidden by overlapping it with urunc's own setup work. before writing any code i checked, across all 6 unikernel types and all 5 hypervisors, exactly what each config step actually depends on:
so the real, honest overlap window i came up with: rootfs prep can run concurrently with the expensive part of network setup (tap device creation, tc rules, fetching interface info), since rootfs prep only needs a cheap early check. everything from concretely: added 4. stopped exec-ing into firecracker for api-basedapi-based needs urunc to stay alive to drive the socket, so it can't just exec-replace itself into firecracker anymore. added one thing i wanted to confirm: whether the child needs pivot_root or chroot for confinement. -- 5. a real bug this surfacedfirst live test of api-based failed immediately: firecracker's traced it to checked (in the vendored 6. how i measured both approachessame methodology for both, to keep the comparison fair:
7. results
api-based came out about 15% faster on average and noticeably more consistent (95 ms spread vs 1050 ms spread) in this first pass. 5 runs each on one machine, one image, one unikernel type, so this is a real signal, not yet a large enough sample to call final. questions
pr with all of the above: #809 |
|
hi @cmainas just reminding you to put the tuesday's sync summary and share the ppt for the lifecycle you did share in meeting. thanks |
|
Hello @Anamika1608 , a few notes from the previous sync:
|
|
hi @cmainas, update on the api-based approach after restructuring it the way you described in the last sync. what i changedpreviously i had misunderstood the parallelization: i was overlapping urunc's own steps with each other (rootfs prep alongside network setup) and still sending the whole vmm configuration in one burst at the very end. now it follows the staged design you outlined:
one implementation detail worth mentioning: the api client now holds a single persistent connection to the socket, established at spawn time. this is because results, 20 runs per modesame methodology for both: timer starts when
it is a dead heat, 2 ms apart on average, with fully overlapping ranges and no failures either side. i want to correct my earlier report explicitly: the "api is about 15% faster" number i shared from 5 runs does not survive a larger sample. that gap came from a single 2279 ms config-file outlier inflating a small sample. why i think the overlap does not show upof the roughly 1250 ms total, about a second is the guest kernel booting and the app becoming ready, which nothing on urunc's side can touch. the host-side work that can actually be overlapped (tap creation, rootfs prep) is in the tens of ms, and that is roughly the same size as the extra cost of driving the boot over the socket instead of one file read (i measured about 29 ms standalone earlier). so the two effects roughly cancel. one overlap i have not tried yetthere is still some unused headroom in the current code. the block device list becomes available right after rootfs prep finishes, which can be before network setup finishes, since the two run concurrently. but right now i send i can implement that split, but i want to be honest that i do not expect it to change the comparison meaningfully. rootfs prep is fast, so the extra overlap window is small, and the total is dominated by guest boot either way. it would be a correctness-of-design improvement more than a performance one. where that leaves the decisionwhat the measurements say so far is that api-based costs nothing in startup time, but also gains nothing, at least for this workload (one image, one unikernel type, linux guest, no block rootfs). its actual value is the live control socket to the running vmm, which is what graceful shutdown and later snapshots need and which questions
|
|
i tried reducing the socket polling time and measured the effect in isolation (a small standalone go app that times only up to state Running, so guest boot does not dominate), 30 runs each:
firecracker's socket is actually ready in about 0.1ms (the busy-loop number), so most of the 10ms wait was just polling-granularity waste. 1ms recovers almost all of it (about 11ms off the average), and the busy loop only saves ~1ms more. i committed the 1ms interval (not the busy loop), since 1ms captures nearly all the benefit while a busy loop would spin the cpu until the socket is ready. to be clear about the scope: this does not make the actual container boot faster. the table above is from the isolated test. in the real run (nerdctl run until the guest answers on http), the total is about 1 second, and almost all of that is the guest kernel booting, which this change does not touch. here are the real end-to-end numbers (nerdctl run until the guest returns http 200), 20 runs each, all in the same session so they are directly comparable.
-- separate PR: configurable control socket for the monitors i also raised the new pr (#841) for the socket path configuration for all three vmms, please do address the comment on this pr as well, there only. |
|
Thank you @Anamika1608 for the quick test and posting the results back. These results rule out any time getting wasted from the polling. |
end-to-end timing script (config-file vs api-based)this is the script i used to measure the end-to-end boot time of the two boot modes: it times from how to run it
[monitors.firecracker]
default_memory_mb = 256
default_vcpus = 1
boot_mode = "api" # or "config-file"then
notes for adapting it
the script#!/bin/bash
set -u
IMG="harbor.nbfc.io/nubificus/urunc/chttp-firecracker-linux-aarch64:latest"
RUNS=20
NAME="fc-timing-test"
declare -a TIMES
for i in $(seq 1 $RUNS); do
sudo nerdctl rm -f "$NAME" >/dev/null 2>&1
T0=$(date +%s%N)
sudo nerdctl run -d --name "$NAME" --runtime io.containerd.urunc.v2 "$IMG" >/dev/null
IP=""
for _ in $(seq 1 500); do
IP=$(sudo nerdctl logs "$NAME" 2>&1 | grep -oP "ipaddr=\K[0-9.]+" | head -1)
[ -n "$IP" ] && break
sleep 0.02
done
if [ -z "$IP" ]; then
echo "RUN $i: FAILED to find guest IP"
sudo nerdctl rm -f "$NAME" >/dev/null 2>&1
continue
fi
READY=0
for _ in $(seq 1 1000); do
CODE=$(curl -s -m 1 -o /dev/null -w "%{http_code}" "http://$IP:80/" 2>/dev/null)
if [ "$CODE" = "200" ]; then
READY=1
break
fi
sleep 0.02
done
T1=$(date +%s%N)
if [ "$READY" -eq 1 ]; then
MS=$(( (T1 - T0) / 1000000 ))
echo "RUN $i: $MS ms (guest IP $IP)"
TIMES+=("$MS")
else
echo "RUN $i: FAILED, guest never responded"
fi
sudo nerdctl rm -f "$NAME" >/dev/null 2>&1
done
echo "---"
if [ "${#TIMES[@]}" -gt 0 ]; then
MIN=${TIMES[0]}; MAX=${TIMES[0]}; SUM=0
for t in "${TIMES[@]}"; do
(( t < MIN )) && MIN=$t
(( t > MAX )) && MAX=$t
SUM=$(( SUM + t ))
done
AVG=$(( SUM / ${#TIMES[@]} ))
printf "%s\n" "${TIMES[@]}" | sort -n > /tmp/times-sorted.txt
N=${#TIMES[@]}
if (( N % 2 == 1 )); then
MEDIAN=$(sed -n "$(( (N+1)/2 ))p" /tmp/times-sorted.txt)
else
M1=$(sed -n "$(( N/2 ))p" /tmp/times-sorted.txt)
M2=$(sed -n "$(( N/2+1 ))p" /tmp/times-sorted.txt)
MEDIAN=$(( (M1 + M2) / 2 ))
fi
echo "successful runs: ${#TIMES[@]}/${RUNS}"
echo "min: ${MIN} ms, max: ${MAX} ms, avg: ${AVG} ms, median: ${MEDIAN} ms"
else
echo "no successful runs"
fi |
|
Thank you @Anamika1608 for sharing the script! |
|
Hello @Anamika1608 , here are a few notes form yesterday's sync:
|
|
hi @cmainas, tested the block setup with the image you gave ( devmapper works, both boot modes. blockfile does not work (and it is not related to my PR). one small docs note for aarch64. so overall: block-based rootfs is verified working for the staged boot, using devmapper. |
|
Hello @Anamika1608 , thank you for the test. The blockfile issue will get resolved with #835 |
|
Let's do it one by one, because every monitor has its own API and the PR will get huge. |
|
hi @cmainas, i have splitted the work across 3 prs -
two questions: 1. custom in config-file mode firecracker is exec'd after one of the fix: spawn firecracker already chrooted ( 2. api-driven boot for qemu / cloud-hypervisor. right now both just expose the socket for post-boot control. cloud-hypervisor's REST api can drive the whole boot over the socket like firecracker's api mode (same parallelization potential); qemu can only trigger the boot over QMP ( |
|
Hello @Anamika1608 , thank you for the PRs
The monitor process must execute inside the chroot environment and the socket must be placed inside the chroot environment for security reasons. The correct handling will be a puzzle because on one hand we want to have parallelization, which requires some operations to take place before chrooting and we also want firecracker process confined in a chrooted environment we create. I was not aware of the chroot capabilities of Go exec. They might be handy here, but we need to properly evaluate it (security wise too).
Yeap, we should use the api-driven boot mode in the same way as firecracker (wherever applicable). Just keep in mind the comment above. |
|
hi @cmainas, update on all three PRs, following your notes on confinement and using the api-driven boot for the other monitors. firecracker #809
qemu #841
cloud hypervisor #847
i am also writing up the VMM lifecycle design document next and will share it when it is ready. |
|
i built and tested the urunc side of graceful shutdown ("half 1"), following the monitor-native direction from @pmoust's last comment. this is the part that sends the shutdown event; teaching urunit to react to it is the separate next step. here is what i did, how i tested it, and two things i want your call on. PR - #869 ( i reached the PR limit, so thats why i've opened it as draft) approachit builds on the config-based control socket (#850). in that mode there is no supervising urunc process, the monitor is the container's init process, so the natural place to press the button is the
it is fire-and-return: urunc does not wait and does not add its own timeout. the container manager already escalates to SIGKILL after its grace period, so there is no reason for urunc to be a second timeout authority. any failure, an unsupported monitor, the feature being off, or any signal other than SIGTERM falls back to the exact kill behaviour we have today. the design decision i want your call onthe feature is an opt-in flag ( Firecracker realityFirecracker only has testingbuilt and ran an 8-case live matrix on aarch64. but the main thing: urunit does not react to these events yet, so no test shows a guest actually cleaning up and shutting down. what the tests prove is that urunc presses the button correctly and returns, and that the fallback is safe. the clearest evidence is the stop latency:
that gap is the button being pressed. other cases: a custom a separate bug i want to flagi also want to pick up @pmoust's finding about urunit and Cloud Hypervisor: urunit exits the VM with i have not reproduced this myself yet (the Cloud Hypervisor guest does not fully boot on my aarch64 setup, so the app-exit path is not something i can exercise here), so i am flagging it as @pmoust's observation rather than something i verified. this is independent of graceful shutdown (it is the normal app-exits path, not the stop-from-outside path). i would like to open it as its own issue rather than fold it in here, since the fix is in urunit and has its own x86-vs-aarch64 considerations. is that ok? questions
|
|
Hello @Anamika1608 , thank you for following the plan and creating a PR based on it. The approach sounds good and it is one of the reasons we want to have the option of exposing the monitor socket. However, we need to take into account the cases where urunc is used as a standalone container runtime and not invoked through the shim. In that case, there will be no other process checking if the container really stopped or not. Therefore, we should keep your approach of using kill, but in delete with Since this feature depends on the existence of a socket for a monitor, we can only use it if the socket exists. Therefore, there is no need for an extra configuration option. Even if Apart form the urunc changes, we will also need to update urunit accordingly. These changes need to be in-sync and through the end-to-end tests we will be able to verify if everything works properly. Let;s create a different issue for cloud-hypervisor. It is easily resolvable and it does not need to depend on these changes. |
|
Hello @Anamika1608 , some TODOs from today's meeting:
Please let me know if I forgot something. |
|
yes, i think we are good. one more thing, that we discussed to check the timeout thing to check if the guest is capable or the it is taking more time. will see this if i can verify that flow |
|
Oh yes, we need to check that too. |
|
hi @cmainas , so i checked this timeout thing for all three monitors, and it is already integrated in the graceful shutdown PR for all three. Timeout: 5 seconds, every monitor. QEMU uses Fallback: one shared path. Signal() calls |
|
hi @cmainas, thank you for providing me the image. the whole qemu flows work end to end. i did test with the image - urunc #869 sent system_powerdown -> the driver kernel delivered KEY_POWER -> our urunit nubificus/urunit#15 read it and SIGTERM'd the app -> the app exited cleanly -> urunit unmounted and stopped the VM. but still the firecracker and cloud hypervisor are not tested, since i am on aarch64 env. |
|
hi @cmainas, i tested snapshot and restore with firecracker and qemu. both work in urunc, live on aarch64. every command goes over the control socket from #850, reached at FirecrackerFirecracker has a clean snapshot API. Snapshot: Restore: copy both files out, stop the container, create the tap QEMUQEMU has no single snapshot call, so i migrate to a file and restore with Snapshot: Restore: the new qemu needs the same command line as the original. urunc builds that itself, so this is easy for the real feature; for the test i rebuilt it by hand (kernel and initrd copied out of the monitor rootfs, The mechanism is proven. The rest is integration:
|
|
Hello @Anamika1608 , a few notes from today's sync:
Please let me know if I forgot anything. |
|
hi @cmainas, so i researched about the containerd support, we can't build this end to end right now. containerd supports the checkpoint but the restore is still not supported in low level.
|
|
Ok, so as far as I understand we should structure our solution around KEP-5823? |
|
yes, correct |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Mentorship plan: Improve lifecycle management of sandbox monitors
This discussion outlines the proposed plan for the CNCF mentorship project "Improve lifecycle management of sandbox monitors”.
The plan is structured into three phases. Each phase has clear goals and specific outcomes, along with suggested tasks and sub-tasks to help guide the work. The listed sub-tasks are meant as guidance and reference points, they are not strict requirements. The exact order of tasks within each phase can be adjusted as long as the main outcomes are achieved.
Phase 1
The goal of Phase 1 is to build the necessary background and get familiar with IPC over sockets in linux namespaces. This phase is expected to last up to 3 weeks (08/06/2026 – 29/06/2026).
Tentative tasks and sub-tasks
Outcome
Deadline: Completed no later than 28/06/2026 (AoE).
Description: A report with the:
Phase 2
The goal of Phase 2 is to extend the quick PoC above to a proper solution which will be integrated in urunc. This phase is expected to last up to 4 weeks (29/06/2026 – 27/07/2026). During this time, the focus will shift to a proper design and implementation of a solution for all monitors
Tentative tasks and sub-tasks
Outcomes
Deadline: Completed no later than 27/07/2026 (AoE).
Description: The following items:
Phase 3
The goal of Phase 3 is to utilize the VMM API for adding support for CRIU. This phase is expected to last up to 4 weeks (27/07/2026 – 31/08/2026). During this time, the focus will shift to utilizing the monitor's API for creating and restoring snapshots and integrating this workflow with CRIU.
Tentative tasks and sub-tasks
Outcomes
Deadline: Completed no later than 31/08/2026 (AoE).
Description: The following items:
All reactions