Cut runner VM boot to guest-ready from ~1.6s to ~0.53s - #5
Open
domenkozar wants to merge 10 commits into
Open
Conversation
Use deadpool Object instead of bb8 PooledConnection in oauth_store, matching the actual DbPool type. Convert eyre::ErrReport to crate Report via .into() in github::serve. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Keep devenv-init as PID 1 and spawn devenv-driver as a child. The pid1 crate inside the driver then no-ops instead of re-executing the 20MB driver binary, which cost a consistent ~100ms per boot. As PID 1, init now reaps orphaned zombies and powers off the VM (ACPI S5, which makes cloud-hypervisor exit) when the driver exits without completing a job. Previously the driver could not halt the VM at all: it drops privileges before calling reboot, so failed VMs hung until killed from the host. Add quiet to the guest kernel cmdline to cut console printk overhead during boot, and poll for the virtiofsd socket every 10ms instead of 100ms when launching a VM. Treat a VM exit without a Complete message from the guest as a job failure. A clean VMM exit code no longer counts as success, since the guest can now power itself off after a failed job. Measured via interleaved A/B launcher runs (13 pairs): boot_vm to guest-ready improved in every pair, mean improvement ~100ms, median 1.60s to 1.48s over the first 8 pairs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The digest dependency was dropped from backend/Cargo.toml but the lock update was not committed, which breaks nix builds that pass --frozen to cargo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bake the devenv-driver binary and its ldd closure into /prewarm-list in the rootfs. devenv-init reads those files with parallel threads right before spawning the driver, racing ahead of the dynamic linker's demand-paged faults over virtiofs, which dominated driver startup. Measured via interleaved A/B launcher runs (8 pairs): boot_vm to guest-ready improved in every pair, median 1.07s to 0.85s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keeps Cargo.nix in sync with the Cargo.lock fix in e1f56e6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The guest boots a kernel directly under cloud-hypervisor with a virtiofs root and virtio devices only, but inherits the full nixpkgs distro config, and kernelPreferBuiltin turns its module drivers into built-ins that register on every boot: DRM, sound, USB, HID, wireless, Bluetooth, Intel ethernet, Hyper-V, SCSI/ATA, RAID and device-mapper, IMA. Force those off. vmlinux shrinks from 19.5MB to 14.4MB. Verified the guest still configures virtio-net and runs the driver to the same point. Measured via interleaved A/B launcher runs (10 pairs): boot_vm to guest-ready improved in every pair, median 0.91s to 0.82s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mitigations=off and init_on_alloc=0 trade guest-internal hardening for speed, which is sound here: each job VM is single tenant and ephemeral, the job already runs arbitrary code inside it, and isolation comes from the virtualization boundary, not the guest kernel. tsc=reliable skips clocksource stability checks, 8250.nr_uarts=0 skips legacy UART probing, random.trust_cpu=on seeds the crng from RDRAND instead of waiting for entropy, and pci=lastbus=0 stops PCI enumeration after bus 0 where cloud-hypervisor puts all virtio devices. Measured via interleaved A/B launcher runs (10 pairs): boot_vm to guest-ready median 0.80s to 0.71s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…esystems The guest console is virtio hvc0, so the legacy 8250 UART driver only adds probing time. DEBUG_WX is a boot-time W+X diagnostic scan. Crypto self-tests run at algorithm registration on every boot. The disk and network filesystems are unreachable behind a virtiofs root with no block devices. Measured via interleaved A/B launcher runs (10 pairs): boot_vm to guest-ready median 0.75s to 0.69s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
linux-resources shipped the compressed bzImage, so the guest paid zstd decompression and legacy setup on every boot even though the kernel already has CONFIG_PVH=y. Ship the debug-stripped vmlinux ELF instead; cloud-hypervisor loads it through the PVH entry point directly. Found via cloud-hypervisor -vv logs: 410ms sat between vCPU start and the first virtio device activation, mostly kernel decompression. Measured via interleaved A/B launcher runs (10 pairs, same kernel as bzImage vs PVH ELF): boot_vm to guest-ready median 0.69s to 0.54s, faster in all pairs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every VM launch copied the 306MB / 10425-file nix store into a per-VM directory shared over virtiofs. The copy is metadata-bound (inode creation plus --preserve=all), costing ~0.9s per launch and dominating end-to-end latency now that boot is ~0.5s. Build the store once at nix-build time as a read-only erofs image (mkfs.erofs --force-uid=1000 --force-gid=100), ship it as a resource artifact like the kernel, and attach it to every VM as a virtio-blk disk. devenv-init mounts it from /dev/vda and overlays a tmpfs upper at /nix/store for job writes. The small mutable rootfs stays on virtiofs. Because ownership is baked into the image, the driver no longer chowns the store recursively (10k files); it only fixes up /nix/var and the overlay root. One image file is also shared across every VM on a host, so the store lands in the host page cache once rather than per VM. Requires EROFS_FS and OVERLAY_FS in the guest kernel. Measured via interleaved end-to-end A/B (8 pairs, launcher start to guest ready): host prep 0.89s to 0.16s, boot 0.54s to 0.45s (virtiofsd no longer serves the store), total 1.43s to 0.62s. Validated: fsck.erofs clean, store bytes identical to the source store, on-disk uid/gid baked to 1000:100, and the guest boots and runs the dynamically-linked driver plus its 90 shared libraries entirely from the erofs-backed overlay. A real build writing to the overlay upper still needs validation on a runner with network. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Cuts end-to-end runner VM launch (launcher start to guest-ready) from ~1.4s to ~0.6s on cherimoya, of which boot_vm to guest-ready drops from ~1.6s to ~0.53s. Measured with interleaved A/B runs (old/new alternating to cancel thermal drift, medians over 8-12 pairs per change).
Changes
Measured dead ends (documented so nobody retries them)
What remains
~100ms driver startup that the static-musl build in cachix/devenv#2967 will mostly remove (and it obsoletes the prewarm list here), the residual ~0.16s host prep (small rootfs copy + virtiofsd spawn), and residual kernel time. VM snapshot/restore (docs/snapshotting.md) is the endgame; the erofs store here shrinks the vhost-user footprint that restore has to survive.
Tuning note
The overlay upper for store writes during a job is tmpfs at size=75%, so large build outputs consume guest RAM. A per-VM scratch disk may be preferable in production; called out for review.
🤖 Generated with Claude Code