Mykhailo Kryshtop

Hundreds of commands and none of them is rg: adding your own tool to an AOSP image

· aospsoongprebuiltsrusttoyboxcuttlefish

Twice a week I type rg into an adb shell by accident, and twice a week the device tells me what it thinks of my muscle memory:

$ adb shell rg --version
/system/bin/sh: rg: inaccessible or not found

What the device does have is one binary with everything in it:

$ adb shell toybox --version
toybox 0.8.11-android
$ adb shell toybox | wc -w
209

209 commands — ls, grep, sed, find, ps, mount, tar, even awk — in 577 KB of /system/bin/toybox, symlinked under every name. Beside it sits toolbox (169 KB) holding the handful of things that are Android-specific rather than POSIX: getprop, setprop, start, stop, getevent, modprobe. The shell is mksh. There is no /bin, no /usr, no package manager, and nothing resembling apt install.

The reason it’s toybox and not busybox is not technical. Busybox has roughly twice as many applets and is the better tool by most measures; it is also GPLv2, and Android’s userspace is kept clear of copyleft obligations on purpose. Toybox is 0BSD — its LICENSE file grants permission to use, copy, modify and distribute “with or without fee”, with no conditions at all. A deliberate product decision, then, not an oversight, and one that no amount of wishing will get you rg through.

So if you want a tool that isn’t in the image, you have three options: push a static binary to /data/local/tmp and live with it being ephemeral; adopt the project into the tree from source, the way AOSP does it; or import a prebuilt binary. I wanted rg in the image of the Cuttlefish device I develop against — flashed, labelled, present on every boot — so this post is about the third route, and about the one thing it quietly takes away from you.

The proper way, and why I didn’t take it

If you want to see a source adoption done right, read external/ethtool. It’s small enough for an afternoon and it’s close to the ideal case:

$ git -C external/ethtool rev-list --count HEAD
1270
$ git log --oneline --no-merges -- Android.bp
cc5ae5c ethtool: turn off pretty print as it causes binary bloat
7b08809 bump version to 6.5
4afca79 bump version to 6.1
...

1270 commits of genuine upstream history, and the Android-side commits are “Merge upstream ethtool v6.5” followed by a one-line touch-up to Android.bp. The entire local delta is build glue. That’s why the project could track v5.4 → 5.6 → 5.10 → 5.15 → 6.1 → 6.5 without a patch queue: nobody ever edited a .c file.

The glue itself is a good read, because every line is a lesson. There’s no config.h — what autoconf would have discovered is passed as -DPACKAGE="ethtool" -DVERSION="6.5". A dependency (libmnl) is vendored wholesale into the same module. Kernel uapi headers are snapshotted under uapi/, because Bionic’s are not glibc’s. Four -Wno- flags appear because AOSP builds external code under a much stricter clang than upstream’s CI. And the source globs are hand-tuned around Soong’s lack of pattern exclusion, with a comment that deserves a plaque:

srcs: [
    "[a-s]*.c",
    "t[a-d]*.c",
    // avoid test-*.c -- note these are shell globs, not regexps
    "t[f-z]*.c",
    "[u-z]*.c",
],

I would happily have done that for ripgrep. Ripgrep is Rust, and Rust changes the shape of the problem entirely.

Soong has no network access and does not run cargo. There is no fetch step at build time, so every crate in a program’s transitive dependency graph must already exist in the tree as a module before the leaf binary can resolve a single rustlibs: entry. AOSP keeps them in external/rust/android-crates-io — 389 crates, managed by a pseudo-crate whose Cargo.toml pins exactly one version of each for the whole platform:

aho-corasick = "=0.7.20"
regex        = "=1.7.3"
regex-automata = "=0.1.10"
bstr         = "=1.3.0"
walkdir      = "=2.4.0"

Now compare that with what cargo build actually pulled for ripgrep 14.1.1:

crate pinned in AOSP 15 ripgrep 14.1.1 wants
aho-corasick 0.7.20 1.1.3
regex-automata 0.1.10 0.4.7
bstr 1.3.0 1.10.0
walkdir 2.4.0 2.5.0

regex-automata 0.1 to 0.4 is not a bump, it’s a different library. And regex is shared with Bluetooth, Keystore, virtualization and everything else in the tree, so raising the pin means fixing every other consumer and not regressing their tests. There’s an extra_versions/ escape hatch for holding a second version of a crate, used sparingly and on purpose.

That’s the real cost of a Rust source import: not vendoring thirty crates, which is mechanical and tool-driven, but the platform-wide version unification behind it. A weekend project turns into a quarter, in exchange for a search tool. So: a prebuilt.

Building it

The pleasant surprise is that you don’t need to install anything. The tree already ships a Rust toolchain with an x86_64-linux-android std:

$ ls prebuilts/rust/linux-x86/1.81.0.u1/bin
cargo  cargo-clippy  cargo-fmt  clippy-driver  rust-analyzer
rust-gdb  rust-lldb  rustc  rustdoc  rustfmt
$ ls prebuilts/rust/linux-x86/1.81.0.u1/lib/rustlib | grep android | grep -v manifest
aarch64-linux-android
armv7-linux-androideabi
i686-linux-android
riscv64-linux-android
x86_64-linux-android

You do need a linker that knows Bionic, and the NDK is the easy answer. (prebuilts/ndk/current in the tree holds only STL sources — no sysroot — so don’t go looking for one there.) The whole configuration is eight lines:

# .cargo/config.toml
[target.x86_64-linux-android]
linker = "$NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/x86_64-linux-android30-clang"
ar     = "$NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar"
rustflags = [
    "-Cpanic=abort",
    "-Clink-arg=-Wl,-z,max-page-size=16384",
]

Both rustflags entries are there because a build refused to proceed without them, and neither is discoverable in advance. Take out the first one and you get:

error: the crate `panic_unwind` does not have the panic strategy `unwind`

AOSP’s prebuilt std for the *-linux-android targets is compiled with panic=abort, so an unwinding build has nothing to link against. A rustup-installed toolchain would not have shown you this; it’s the price of borrowing the platform’s. The second entry has a section to itself further down — it is the reason my first build was rejected.

With that, cargo build --release --target x86_64-linux-android and llvm-strip produce exactly what you want:

$ file x86_64/rg
ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically
linked, interpreter /system/bin/linker64, for Android 30, built by
NDK r23b (7779620), stripped
$ readelf -d rg | grep NEEDED
 (NEEDED)  Shared library: [libdl.so]
 (NEEDED)  Shared library: [libc.so]

Two Bionic libraries and nothing else — Rust static-links its std, and ripgrep’s default feature set has no C dependencies. adb push it to /data/local/tmp, confirm it runs, and the hard part is over. Or so I thought.

Six files and a module type

Into the tree it goes, in a vendor directory of my own:

vendor/kryshtop/prebuilts/ripgrep/
├── Android.bp
├── METADATA              # upstream URL, version, licence type
├── MODULE_LICENSE_MIT
├── LICENSE-MIT, UNLICENSE
├── README.android        # the exact build command, and why
└── x86_64/rg

The module type for an imported executable is cc_prebuilt_binary, and it doesn’t care what language produced the ELF:

cc_prebuilt_binary {
    name: "rg",
    arch: {
        x86_64: { srcs: ["x86_64/rg"] },
    },
    compile_multilib: "64",
    // Rust static-links its std; only Bionic is dynamic here.
    shared_libs: ["libc", "libdl"],
    // Already stripped by llvm-strip at import time.
    strip: { none: true },
}

shared_libs on a prebuilt looks redundant — nothing is being linked — but it’s a declaration that gets verified: Soong checks that every DT_NEEDED entry in the binary appears in that list. The license {} block above it (omitted here) names SPDX-license-identifier-MIT and SPDX-license-identifier-Unlicense and points at the two licence files.

README.android is the file nobody makes you write and everybody wishes past-them had. Mine records the exact cargo invocation, the NDK version, why -Cpanic=abort is there, and why this is a prebuilt rather than a source import. A binary in a source tree is an assertion that somebody, somewhere, could rebuild it; that file is the only thing making the assertion true.

The check that caught me

Fifty-six percent into the build:

FAILED: .../obj/EXECUTABLES/rg_intermediates/check_elf_files.timestamp
rg: error: Load segment has alignment 4096 but 16384 required.
rg: note: Fix suggestions:
rg: note:   use linker flag "-Wl,-z,max-page-size=16384" when compiling this lib
rg: note: If the fix above doesn't work, bypass this check with:
rg: note:   Android.bp: ignore_max_page_size: true,
rg: note:   Android.mk: LOCAL_IGNORE_MAX_PAGE_SIZE := true
rg: note:   Device mk: PRODUCT_CHECK_PREBUILT_MAX_PAGE_SIZE := false

Three of those four suggestions are ways to make the message go away, and at 56% of a build they are extremely tempting. They are also all wrong here. Android 15 expects userspace binaries to tolerate a 16 KB kernel page size; segments aligned to 4 KB cannot be mapped by a kernel whose pages are 16 KB, so suppressing the check ships a binary that simply won’t load on such a device. The default isn’t arbitrary either — build/make/core/config.mk picks it deliberately:

else
  # The default binary alignment for userspace is 16384.
  TARGET_MAX_PAGE_SIZE_SUPPORTED := 16384
endif

The first suggestion is the real fix — and for a Rust build it lands in the same rustflags array as the panic strategy, which is why the cargo config near the top of this post has two entries in it:

"-Clink-arg=-Wl,-z,max-page-size=16384",
$ readelf -lW x86_64/rg | awk '/LOAD/{print $NF}' | sort -u
0x4000
source import (external/ethtool) prebuilt import (vendor/…/ripgrep) external/<tool>/ upstream git merged in; local delta = Android.bp cargo build --target x86_64-linux-android built outside the tree, NDK linker, then stripped cc_binary { … } Soong compiles and links it cc_prebuilt_binary { … } Soong only copies it global link flags apply -Wl,-z,max-page-size=16384 cc/config/x86_64_device.go — for free no link step here the same flags are now yours to pass, by hand, in someone else's build system check_elf_file every DT_NEEDED declared in shared_libs? load segments aligned to TARGET_MAX_PAGE_SIZE_SUPPORTED (16384)? /system/bin/rg plus its attribution in /system/etc/NOTICE.xml.gz
The two routes, and the box that’s missing from the right-hand one. A cc_binary is linked by Soong, so it inherits the platform’s global linker flags without anyone thinking about it. A cc_prebuilt_binary is only copied — so the same guarantees have to be reproduced by hand, in a different build system, by someone who has to know they exist. check_elf_file is the net under that gap.

Here’s why I think this is the most useful twenty minutes of the whole exercise. That linker flag is not something an AOSP developer normally types. It’s in build/soong/cc/config/x86_64_device.go:

pctx.VariableFunc("X86_64Lldflags", func(ctx android.PackageVarContext) string {
	maxPageSizeFlag := "-Wl,-z,max-page-size=" + ctx.Config().MaxPageSizeSupported()
	...
})

Every binary Soong links gets it for free, along with a long tail of other global flags nobody thinks about. Import a prebuilt and you skip the link step — and with it, silently, every one of those defaults. You haven’t just moved a build off-tree; you’ve taken personal ownership of a set of platform ABI guarantees you probably can’t enumerate.

check_elf_file exists precisely because that’s a bad position to be in. It is not bureaucracy standing between you and your binary; it’s the only thing checking the work the linker would have done for you. The right response to it is to fix the binary, and the ignore_ knobs are for vendor blobs you cannot rebuild — not for the one you produced ten minutes ago and can rebuild in eight seconds.

Into the product, properly

Two lines in a product makefile — mine go into Cuttlefish’s device/google/cuttlefish/shared/device.mk. The pattern is already in the tree if you want a precedent to point at: shared/minidroid/device.mk carries its own PRODUCT_PACKAGES_DEBUG block.

# Developer convenience tool, prebuilt.
# See vendor/kryshtop/prebuilts/ripgrep/README.android.
# _DEBUG so it lands on eng/userdebug only and never on a user build.
PRODUCT_PACKAGES_DEBUG += \
	rg

PRODUCT_PACKAGES_DEBUG is the variable people reach for ifeq to emulate. build/make/core/main.mk sets tags_to_install := debug for userdebug and debug eng for eng, and leaves it empty for user; PRODUCT_PACKAGES_DEBUG is only consulted when the debug tag is present. One variable, no conditionals, and the tool cannot escape into a production image by accident.

And then the part that’s easy to skip: attribution. Because the license {} module named its kinds and texts, Soong generated the metadata for the module —

module_name: "rg"      module_types: "cc_prebuilt_binary"
license_kinds: "SPDX-license-identifier-MIT", "SPDX-license-identifier-Unlicense"
license_conditions: "notice", "unencumbered"
license_texts: vendor/kryshtop/prebuilts/ripgrep/{LICENSE-MIT,UNLICENSE}
installed: .../system/bin/rg

— and ripgrep’s MIT attribution now appears in the image’s system/etc/NOTICE.xml.gz, which is what the device shows under Settings → About → Legal information. Two lines of Blueprint, and the difference between shipping a licence notice and shipping a violation.

What the device says

$ adb shell ls -lZ /system/bin/rg
-rwxr-xr-x 1 root shell u:object_r:system_file:s0 4487752 /system/bin/rg
$ adb shell rg --version
ripgrep 14.1.1 (rev 4649aa9700)
$ adb shell rg --no-heading -n "class late_start" /system/etc/init/
/system/etc/init/usbd.rc:2:    class late_start
/system/etc/init/update_engine.rc:3:    class late_start
/system/etc/init/traced_perf.rc:24:    class late_start

No sepolicy work was needed, incidentally: files installed to /system/bin get system_file by default and the shell domain may execute those. Had I installed to /vendor/bin instead, that would have been a different afternoon.

And a number that made me laugh: rg is 4,487,752 bytes. The toybox binary it’s sitting next to — all 209 commands of it — is 577,144. The search tool is roughly eight times the size of the entire command-line environment it just joined. On a device where images are budgeted in kilobytes, that alone is a decent argument for PRODUCT_PACKAGES_DEBUG.

The bill

The honest costs, so nobody discovers them later:

It isn’t reproducible. Nobody can rebuild that binary from the tree. This is exactly why AOSP proper prefers source imports, and why README.android is doing real work rather than being documentation theatre.

Licence duties differ by route. MIT and the Unlicense are the easy case: ship the notice, done. Had ripgrep been GPL, shipping a binary would have created a source-delivery obligation that a source import wouldn’t — the opposite of most people’s intuition about which route is “safer”.

It’s a blob in git. 4.5 MB per architecture, in a source repository. Fine for a personal tree; a shipping product wants that in a dedicated prebuilts project.

Reading list

Everything here is in an android-15.0.0_r36 tree.