I was wrong about the sandbox: strace vs. my favorite theory

10 Aug 2026 · aospcuttlefishcrosvmapparmorseccompdebugginglinux

In the previous post I diagnosed why Cuttlefish’s crosvm dies on my Ubuntu 26.04 builder: the seccomp syscall allowlists shipped with android-15.0.0_r36 predate glibc 2.42, a jailed device child makes a syscall the frozen policy doesn’t know, SIGSYS, dead, “Connection reset by peer”. I called the sandbox a liability on bleeding-edge dev boxes and turned it off.

The theory was coherent, matched a real failure class, explained every observation I had — and was wrong. This post is the disproof, the actual root cause, and what it changes about the conclusions. I’m leaving the original up as-is; being wrong in public is only useful if the correction is public too.

The experiment that was supposed to confirm it

The glibc theory implied a clean fix that I wanted anyway: run the whole Cuttlefish host stack in a Debian 12 container. Old glibc (2.36) never emits the new syscalls, so the frozen allowlist holds; the host kernel stays shared, which is fine because kernels happily run old syscalls forever. As a bonus, each AOSP version gets its own userspace era instead of me chasing host-OS compatibility per tag.

That’s not a hack, it turns out — it’s the blessed path. The android-cuttlefish repo’s own package-build container (tools/buildutils/cw/) is already Debian 12, so the debs I’d fought to build on the bare host rebuilt in it without a single one of my patches mattering (the host-built ones needed GLIBC_2.43 and were unusable anywhere else — worth checking with objdump -T | grep GLIBC before shipping debs built on a fresh distro). Runtime image: upstream’s Containerfile with the base swapped to debian:12, AOSP out/ bind-mounted in, launch as uid 1000 so the mounts stay user-owned.

The prediction was explicit: same crosvm binary, same kernel 7.0, glibc 2.36 instead of 2.42 → sandbox works.

crosvm died in one second. Same “failed to create a PCI root hub: failed to create proxy device”, same connection reset. On glibc 2.36.

That’s the moment a theory earns deletion: it finally made a falsifiable prediction and the prediction failed. Whatever killed the children didn’t care which glibc asked. The only common factor left was the machine itself.

strace doesn’t theorize

I’d inferred the SIGSYS instead of observing it — audit logging was off, so the seccomp kill would have been invisible, and the failure shape fit. Time to actually watch. The crosvm in out/host/linux-x86/bin/ is a two-line wrapper script around the real binary, which makes instrumentation trivial:

exec strace -ff -o /tmp/cf-strace/crosvm \
  "$(dirname "$0")/$(uname -m)-linux-gnu/$(basename "$0")" "${@}"

Relaunch, collect the dying child’s trace:

write(2, "crosvm[1]: libminijail[1]: unsha"..., 80) = 80
unshare(CLONE_NEWNS)  = -1 EPERM (Operation not permitted)
...
tgkill(1, 1, SIGABRT) = 0
--- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL} ---
--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} ---
--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} ---

Three revelations in ten lines:

1. The killer wasn’t seccomp. The child never lived long enough to install a filter. It died building the jail itself: unshare(CLONE_NEWNS) — minijail creating the mount namespace it needs to pivot_root the device process into /var/empty — returned EPERM.
2. The child chose to die. After the failed unshare, minijail calls abort(). An error path, not a murder.
3. And then it couldn’t. The child is pid 1 of its own pid namespace, and an init process ignores default-action signals — including its own SIGABRT. The kernel falls back to injecting SIGSEGV (si_code=SI_KERNEL), which also can’t kill it, forever. The trace held 34 million SIGSEGV deliveries — 6 GB of a process failing to die. That unkillable zombie is what the parent experiences as a hung control tube, and finally reports as “Connection reset by peer”.

So the original post’s error-chain reading was right — the message really was four causal layers from the cause. It just had the wrong cause at the end of the chain.

The real culprit

Why does unshare(CLONE_NEWNS) fail for a process that does the same thing successfully on every Google-supported distro?

$ sysctl kernel.apparmor_restrict_unprivileged_userns
kernel.apparmor_restrict_unprivileged_userns = 1

Since 24.04, Ubuntu denies unprivileged user-namespace creation unless the binary carries an AppArmor profile granting userns — that’s why apparmor_parser loads exception profiles named chrome and firefox at boot. Minijail’s jail setup is exactly the dance this restriction targets: create a user namespace to gain CAP_SYS_ADMIN inside it, then unshare the mount namespace. crosvm ships no exception profile, so: EPERM, abort, zombie, connection reset.

The verification needs no Cuttlefish at all:

$ unshare --map-root-user --mount true
unshare: write failed /proc/self/uid_map: Operation not permitted

And it explains the failed container experiment perfectly: the sysctl is kernel-global, and a container swaps userspace, not the kernel. The container user was exactly as unprivileged as I was. My glibc fix never had a chance to matter — though the container test was still the most valuable thing I did, because its failure is what killed the wrong theory.

The fix

sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
echo kernel.apparmor_restrict_unprivileged_userns=0 \
  | sudo tee /etc/sysctl.d/99-cuttlefish-userns.conf

This restores the upstream kernel default — which is what Debian ships, which is why Cuttlefish “just works” on Google’s supported distros and never worked on my box, bare or containerized. After that, launch_cvd boots with the sandbox on: every crosvm device child shows Seccomp: 2 in /proc/<pid>/status, adb attaches, and --enable_sandbox=false retires. (One in-container quirk: pass --enable_sandbox=true explicitly there — the auto-detection quietly resolves to off inside a container, and a silently disabled sandbox is exactly the kind of thing this saga teaches you to verify rather than assume.)

Two notes on doing this responsibly:

• This does not disable AppArmor. Every profile still confines what it confined before. It flips one Ubuntu-specific policy about who may create user namespaces.
• The restriction exists for a reason: user namespaces expose kernel attack surface (netfilter and friends) to unprivileged local code, and real privilege escalations have used them. On a single-user dev builder the trade is easy. On a multi-user Ubuntu host, the surgical fix is a per-binary AppArmor userns exception profile for crosvm, Chrome-style, instead of the sysctl. And user namespaces are the tension in miniature: simultaneously an attack surface and the primitive every modern sandbox is built from. Production Cuttlefish fleets sidestep the whole question — they run Debian, where the upstream default already allows this, or run the orchestration container as root, where the restriction doesn’t apply.

Reconciling with the previous post

What survives, what dies:

• “The failure mode is maximally misleading” — survives, and doubled. Each layer reported the layer below truthfully and uselessly; the chain was just one link longer than I claimed.
• “Read errors as chains, not messages” — survives; I stopped one layer too early, at the first cause that fit a known story.
• “Pinned userspace + rolling kernel = seccomp time bomb” — dies here, as this bug. The failure class is real (frozen allowlists do expire), but it wasn’t this failure, and I asserted it without catching a single SIGSYS. The sandbox, run on a kernel that lets it start, works fine years ahead of its policies.
• “The sandbox isn’t worth keeping on a dev box” — dies. The sandbox was never broken; the platform wouldn’t let it start, and the fix costs one sysctl. It’s back on, everywhere, including the container.

What I’m taking away, round two

• A theory that explains everything you’ve seen is still just a theory. Mine fit every observation, matched a known failure class, and had a satisfying villain. It died the first time it made a testable prediction. If your diagnosis hasn’t survived an experiment designed to falsify it, it hasn’t survived anything.
• strace beats inference. One trace replaced two days of theory with the literal failing syscall and its errno. When a child process dies wordlessly, strace -ff on the parent is the shortest path to truth — and wrapper scripts make it a one-line install.
• Containers don’t contain the kernel. Anything living in a sysctl, an LSM, or the kernel itself follows you into every container on the host. Swapping userspace can’t fix it — but it can test it, which is worth just as much.
• Know your distro’s hardening. apparmor_restrict_unprivileged_userns breaks software that builds its own sandbox — browsers, container runtimes, minijail — with errors that look like bugs in that software. EPERM from anything namespace-flavored on Ubuntu 24.04+: check the sysctl before reading any code.
• pid-namespace init processes die weird. A jailed child that can’t deliver its own SIGABRT becomes an unkillable SIGSEGV loop. If a sandboxed process hangs instead of crashing, “pid 1 can’t take default-action signals” belongs on the suspect list.
• Publish corrections at the same volume as claims. The previous post had the diagnosis wrong and the debugging lesson right. This one exists so the record has both.