From the plan to a working build. cd9660 + mkuzip-compressed UFS + gunion(8) writable overlay + init_chroot pivot. Revision 3: implemented, tested with KDE Plasma 6, with a clear-eyed view of what we'd change next.
reboot -r. Abandoned after kernel source review showed vfs_unmountall(MNT_FORCE) would orphan vnode-backed mds during reroot.
Rev 2 — preload + gunion + reboot -r via loader md_image. Abandoned due to UEFI staging-area limits + heavy RAM cost (entire compressed rootfs in kernel memory forever).
Rev 3 (current) — cd9660 stays as kernel root, vnode-mount uzip, gunion overlay, init_chroot pivot. Working in CI.
login: prompt — which by definition means every prior stage of the boot succeeded.init_chroot. cd9660 stays mounted forever; only decompressed pages of accessed uzip data live in RAM.LIVE_HEADROOM, default 1 GiB). On a host with 16 GiB RAM, the live system still reports just 1 GiB free. Doesn't scale to host capacity. Three paths to fix below.The "obvious" plan was preload rootfs.uzip via the loader and pivot via reboot -r. Source review of sys/kern/vfs_subr.c:5087 and sys/kern/kern_shutdown.c:543-633 showed the failure modes:
kern_reroot() calls vfs_unmountall() with MNT_FORCE — orphans any vnode-backed md whose backing file lives on a now-unmounted filesystem.md_image. UEFI loader's staging area defaults to 64 MiB; expansion via BS->AllocatePages failed under OVMF at ~300 MiB total preload.init_chroot sidesteps both: kernel mount table is unchanged (cd9660 stays mounted), nothing goes through the loader's staging area beyond the kernel itself, and only accessed pages of the uzip decompress into the page cache. This matches what Linux livecds do (cd9660 stays mounted, squashfs is loop-mounted by initramfs scripts, switch_root is a mount-namespace op rather than a kernel reroot).
From sbin/init/init.c:326-336:
if (init_script kenv set)
run_script(...) // runs synchronously, blocks until child exits
if (init_chroot kenv set) // re-read AFTER the script returns
chroot(...)The script can kenv init_chroot=/sysroot before exiting, and init reads that kenv on the literal next line. Match-perfect with helloSystem's geom_rowr branch trick. This is the only in-kernel "switch root" mechanism on FreeBSD that's workable without going through reboot -r's destructive unmount path. (FreeBSD has no pivot_root or switch_root equivalent — verified by grep across sys/kern/vfs_mount.c and sys/sys/mount.h.)
| Build variant | Content | Lower UFS | rootfs.uzip | ISO total | Build time |
|---|---|---|---|---|---|
| Minimal base (no pkglist) | 803 MiB | 1.8 GiB | 230 MiB | 396 MiB | ~7 min |
| + xorg + KDE Plasma 6 | 10.3 GiB | 11.3 GiB | 3.7 GiB | 3.8 GiB | ~36 min |
3.7G rootfs.uzip ← 99% of ISO size
19M rescue ← static busybox-equivalent
16M boot ← kernel.gz + 8 essential modules
3.0M lib ← libs for dynamic /sbin/geom
121K libexec ← ld-elf.so.1 + helpers
38K sbin ← /sbin/geom (dynamic)
... ← init.sh, rootfs.bytes, sysroot/, EFI/| Phase | Time | What dominates |
|---|---|---|
| Runner + vmactions cold start | ~3 min | downloading + booting FreeBSD VM image |
| pkg install xorg + kde | ~4 min | downloading 705 packages, ~2 GiB |
| makefs UFS at 11.3 GiB | ~2 min | writing zeros to disk |
| mkuzip zstd-19 + dedup | ~18 min | compressing 10 GiB real bytes — the bottleneck |
| cdroot prep + ISO + upload | ~3 min | cd9660 makefs + 4 GiB artifact upload |
| Boot smoke-test (KVM) | ~13 min | longer because more pages to fault in |
init_chroot is FreeBSD's closest equivalent and works for our use case.pivot_root or switch_root equivalent. Verified by source grep. init_chroot + chroot is the workaround. cd9660 stays kernel-/ forever; the chroot just hides it from userland.| Symptom | Root cause | Fix |
|---|---|---|
Loader: cannot open /boot/lua/loader.lua | Cherry-picked specific files into cdroot/boot; missed Lua loader scripts | Copy whole /boot tree (now: only what's loader-needed, see below) |
qemu: iothread mutex assertion in TCG | qemu 8.x bug with FreeBSD long-mode transition under TCG-SMP | Use -accel kvm if /dev/kvm exists; -machine q35 as TCG fallback |
OVMF: BdsDxe: failed to load Boot0001 | El Torito EFI image was FAT12; UEFI spec requires FAT16+. Also OVMF prefers iso9660 fallback path | FAT16 ESP at 32 MiB + stage /EFI/BOOT/BOOTX64.EFI on cd9660 root |
Kernel: panic: no init | Forgot /sbin/init + /rescue/ on cd9660 root after switching to cd9660-as-kernel-root | Stage /rescue (tar pipe — see below) and /sbin/init -> /rescue/init on cd9660 |
Kernel: Failed to load kernel 'kernel' | Gzipped kernel saved as /boot/kernel/kernel without .gz extension; loader's gzipfs triggers on suffix | Save as /boot/kernel/kernel.gz |
Mountroot: Mounting from ufs:/dev/md0 failed with error 2 | mfsroot was gzipped without .gz extension; kernel saw raw gzip bytes as md0 | Don't gzip mfsroot (or rename properly) |
init.sh: geom: Unknown command: create | /rescue/geom is statically linked, can't dlopen the union class library | Ship the dynamic /sbin/geom + libs via ldd + /lib/geom/geom_union.so |
init.sh: mdconfig: ioctl(/dev/mdctl): EDOM | Byte-precise size with ${BYTES}b suffix wasn't 4 KiB-aligned (modern md sectorsize) | Round to MiB units: ${MB}m |
geom union: Upper provider md1 size too small, needs N | gunion needs upper ≥ lower × ~1.06 for its bitmap/metadata header | Size upper at max(50% RAM, lower × 1.1 + 64 MiB) |
/etc/rc: mount_cd9660: /dev/iso9660/LIVECD: Invalid argument | Inside chroot, mount -uw / consults kernel's mount table which still has cd9660 at /, not the chroot's view of / | root_rw_mount="NO" in rc.conf — gunion union is already r/w |
Boot test: hangs after SMOKE_TEST_DONE | expect's send "\x01"; send "c" to enter qemu monitor; we use -serial stdio not -serial mon:stdio | close; wait; exit 0 — SIGHUPs qemu via closing spawn |
3.1 GiB ISO with 2.8 GiB /rescue | cp -aR on FreeBSD doesn't preserve hardlinks (unlike GNU cp). /rescue's ~200 hardlinks to one crunchgen binary all expanded to full copies. | tar pipe: (cd src; tar cf - rescue) | (cd dst; tar xf -) |
makefs: minsize rounded up to ffs bsize exceeds maxsize | makefs's default 32 KiB block size requires the requested image size to be bsize-aligned | Round LOWER_BYTES up to 1 MiB boundary |
Build fails at makefs: cannot fit content into requested size | Fixed-LIVE_DISK_SIZE doesn't scale with package additions — adding KDE blew past the 2 GiB ceiling | Switch to LIVE_HEADROOM model: lower = content + headroom, content auto-detected |
Boot: init 16 - - login_getclass: unknown class 'daemon' | base.txz ships /etc/login.conf but not the compiled /etc/login.conf.db; bsdinstall normally builds it during install, which we skip | Run cap_mkdb /etc/login.conf + pwd_mkdb -p on the rootfs after extraction |
| Boot test never sees marker, even though the boot succeeded | Used "Starting local daemons:" marker — that line only prints if /etc/rc.local exists. We deleted rc.local for a clean live boot, so the line vanished too. | Switch to the login: getty prompt as the marker. It always prints on a successful boot and is the strongest "system is fully up" signal. |
/boot/kernel/ has ~80 .ko files. We need maybe 8 at boot; the rest are inside the rootfs.uzip already and reachable via post-chroot kldload. Trimming carrier-side modules saves hundreds of MiB.kernel.gz. ~50 MiB → ~15 MiB.*.symbols files — kernel debug symbols, often as big as the .ko files themselves.kde5 to kde (version-agnostic).kern.evdev.rcpt_mask is informational, not an error. One of those FreeBSD packages-with-helpful-output things.Current architecture works. Three paths to optimize further. None are mandatory; all are real tradeoffs.
Build: makefs -s "$content + LIVE_HEADROOM" bakes empty space into the lower UFS. Runtime: gunion overlay backs writes with swap-md upper. df reports the bake-in headroom as free space.
| Pros | Cons |
|---|---|
|
|
Tighten the lower (no LIVE_HEADROOM at build time). At boot, create gunion with an explicit larger size, then growfs to extend the UFS to fill it. The "extra" space lives in the swap-md upper.
build: makefs -s "${content_size}" # tight, no headroom
mkuzip ...
boot: mdconfig -t vnode -f /rootfs.uzip -u 0
TARGET_MB=$(( $(sysctl -n hw.physmem) / 1048576 / 2 ))
mdconfig -t swap -s "${TARGET_MB}m" -u 1
geom union create -s "${TARGET_MB}m" md1 md0.uzip
growfs /dev/md1-md0.uzip.union # extend UFS to fill the union
mount /dev/md1-md0.uzip.union /sysroot| Pros | Cons / risks |
|---|---|
|
|
Recommended next experiment. Smallest-risk way to get RAM-scaled apparent disk size while keeping gunion's stability.
Keep gunion tight (LIVE_HEADROOM=0 or small). In init.sh, before chroot, mount tmpfs over /sysroot/tmp, /sysroot/var/tmp, /sysroot/var/log, /sysroot/home, /sysroot/root. These dirs get RAM-scaled scratch space.
mount /dev/md1-md0.uzip.union /sysroot
mount -t tmpfs -o mode=1777 tmpfs /sysroot/tmp
mount -t tmpfs -o mode=1777 tmpfs /sysroot/var/tmp
mount -t tmpfs tmpfs /sysroot/var/log
mount -t tmpfs tmpfs /sysroot/home
mount -t tmpfs tmpfs /sysroot/root| Pros | Cons |
|---|---|
|
|
Easy add-on regardless of which other path we take. Worth doing.
Drop gunion entirely. Mount uzip-backed UFS read-only at /lower, tmpfs at /upper, unionfs at /sysroot. File-level overlay; df reports tmpfs's free space (RAM-scaled).
mdconfig -t vnode -o readonly -f /rootfs.uzip -u 0 # md0.uzip
mount -t ufs -o ro /dev/md0.uzip /lower
mount -t tmpfs tmpfs /upper
mount -t unionfs -o below /upper /lower
# /lower is now the merged view; mount it at /sysroot via nullfs or move| Pros | Cons / risks |
|---|---|
|
|
The architecturally cleanest answer if it's stable enough. NomadBSD's specific deadlock trigger (reroot under load) doesn't apply to us. Worth a CI experiment on a branch.
FUSE in the boot path is unwanted. Closes the option even though NomadBSD ships it in production. Listed for completeness.
The 2023 GSoC squashfs-on-FreeBSD effort exists as a branch but never merged. Reviving it is weeks-to-months of kernel work. Even if done, it doesn't solve the writable-overlay problem (squashfs is also block-vs-filesystem mismatched against gunion). Real win would be ~10% smaller ISO from squashfs's tail-packing — not worth it.
Rather than picking one architecture in this repo and discarding the alternatives, the plan is to build them as parallel repos with the same build pipeline and let the architectures be compared head-to-head:
| Repo | Architecture | Status |
|---|---|---|
freebsd-livecd-gunion | cd9660 + uzip + gunion (block-level overlay) + init_chroot. LIVE_HEADROOM baked into the lower UFS at build time. | working, v0.1 |
freebsd-livecd-unionfs (planned) | cd9660 + uzip + in-kernel unionfs + tmpfs upper + init_chroot. RAM-scaled apparent disk size. File-level overlay; matches Linux livecd semantics. | planned |
Same build script structure, same pkg pipeline, same boot-test harness. The two will diverge only in the overlay layer (init.sh + cdroot's overlay-tooling files). Real measurements from each will let us compare ISO size, build time, runtime memory, and stability rather than guessing.
| Option | Apparent FS size scales with host RAM? | Risk | Effort |
|---|---|---|---|
| Status quo (gunion + LIVE_HEADROOM) | No | none | 0 (current) |
| A: gunion + growfs | Yes | medium (combination unverified) | ~5 lines init.sh |
| B: tmpfs overlays for /tmp /home /var/tmp | Partial (those dirs only) | low | ~5 lines init.sh |
| C: in-kernel unionfs | Yes | medium-high (stability) | ~15 lines init.sh + cdroot changes |
| A + B together | Yes (full + scaled scratch) | medium | ~10 lines init.sh |
Try Path A on a branch. Smallest delta from current architecture; biggest user-visible win (RAM-scaled apparent disk). If gunion -s LARGER + growfs compose cleanly in CI's boot smoke-test, merge. If not, the branch is throwaway.
Skip Path C unless Path A fails. The unionfs stability risk isn't justified by the marginal additional gain (file-level granularity vs block-level — practical difference is small).
github.com/pkgdemon/freebsd-livecd-unionfskern_reroot(), why reboot -r is destructiveunmount_or_warn() uses MNT_FORCEg_md_init() generic preload loop (not used in current arch but verified working)git log --since="2 years ago" for active deadlock/race fixes-A zstd -C 19 -d is the right tuningGenerated 2026-05-05. Updates revision 2 with measurements from the working CI build (run 25349540268, minimal, 396 MiB ISO; run 25350809866, KDE, 3.8 GiB ISO) and the architectural options surfaced during implementation. The plan is now an "implementation report + remaining options" document; the architecture section is now a "what's actually working" rather than "what we plan to build". The open architectural questions (RAM-scaled apparent free space) have three concrete paths forward, none of which are required for v0.1.