freebsd-livecd-gunion-reroot — experimental architecture DEFERRED

A research-grade attempt to fix the kernel-vs-userspace namespace split via gunion (block-level RAM overlay) + reboot -r reroot. The simpler symlink trick (cdroot /boot/firmware → /sysroot/boot/firmware) verified working on real hardware, which addresses the immediate firmware-loading problem without an architectural rewrite. This experiment is preserved as a reference for the alternative architecture; pursue if the symlink approach starts hitting limitations or if a future feature requires kernel and userspace to genuinely share a root.

1. Goal & non-goals

1.1 Goal

Build a livecd variant where the kernel's root mount IS the writable rootfs (via gunion + reroot), so kernel-context filesystem operations (firmware loading, kld auto-search, etc.) see the same view as userspace. Specifically: iwlwifi/i915kms/amdgpu firmware loads correctly without needing a 1GB cdroot bloat or a `/boot/firmware → /sysroot/boot/firmware` symlink workaround.

1.2 Non-goals (this experiment)

• No replacement for existing livecd variants in production. If this works, we evaluate whether to retire freebsd-livecd-gunion in favor of it; until then both coexist.
• No optimization for low-RAM systems. The reroot architecture inherently needs the rootfs in RAM (or some non-cd9660-backed location). Machines with <3GB are out of scope.
• No kernel patches. We use FreeBSD's existing kern_reroot(), mdconfig(8), gunion(8), loader(8) features. If any of those need bug fixes to make this work, file upstream tickets, don't carry patches.
• No fancy GUI installer flow. Same boot-test-in-qemu cycle as the other livecd projects.
• No pivot to mfsBSD entirely. The "use a memdisk for the rootfs" trick comes from mfsBSD, but we keep gunion as the writable overlay (not pure tmpfs root). This preserves the layered-RO-with-RAM-overlay model.

2. The problem we're solving

All three of our existing livecd projects share the same architectural shape, and it has the same kernel-firmware-loading bug.

The bug: kernel-context vn_open in sys/kern/subr_firmware.c's try_binary_file() resolves /boot/firmware/<name> against the kernel's root namespace, which on these livecds is the cd9660 mount. Userspace processes (post-chroot) see /boot/firmware/ in the unionfs/gunion view, which has the pkg-installed firmware files. The kernel's view is the cd9660 root, which contains only the bootloader-stage firmware (a small subset). Result: iwlwifi-8000C-36.ucode exists for userspace but not for kernel; wifi doesn't come up.

Verified empirically on a Lenovo laptop running freebsd-launchd's continuous-release ISO: file present (2.4MB) via ls /boot/firmware/iwlwifi-8000C-36.ucode; iwlwifi attaches at the bus level but reports "could not load binary firmware /boot/firmware/iwlwifi-8000C-36.ucode either" followed by "File size way too small!" — the kernel's read returned zero bytes despite the file being right there from userspace.

3. The proposed architecture

The fix is conceptually simple: make the kernel's root mount the same thing userspace sees. chroot can't do this (chroot only changes the userspace process's view, kernel root stays put). kern_reroot() can — it actually unmounts the kernel's root and re-mounts a new one. After reroot, kernel namei resolves paths against the new root.

Three architectural improvements over the chroot variants:

• Kernel namespace = userspace namespace. No split. Firmware loads. Future kernel-context filesystem operations (kld_load search paths, etc.) see the right view.
• Rootfs is in RAM, not on cd9660. Reads are faster (no decompression hit on every page-fault into uzip-backed pages). Trade-off: higher RAM floor.
• cd9660 can be fully unmounted post-reroot. Frees the optical/USB device for other use (uncommon but a valid operation).

4. The busy-cd9660 problem and how to dodge it

The naive version of this — set up gunion + reroot without preloading the rootfs.uzip — fails because:

1. /init.sh runs in cd9660 root
2. mdconfig -a -t vnode -f /rootfs.uzip   ← holds /rootfs.uzip vnode busy
   /dev/md0 is now backed by the FILE on cd9660
3. gunion create /dev/md0.uzip            ← /dev/gunion0
4. reboot -r with vfs.root.mountfrom=ufs:/dev/gunion0
5. kern_reroot() → vfs_unmountall()
6. unmount cd9660 → EBUSY  (md is holding /rootfs.uzip vnode)
7. cd9660 stays mounted; vfs_mountroot() runs anyway with old root still half-mounted
8. Inconsistent rootvnode state → panic in vfs_mountroot() OR
   half-rerooted system in unstable state

The exact failure mode (panic vs hang vs error) depends on FreeBSD's vfs_mountroot implementation details I haven't traced exactly. Either way it's bad enough that "don't try this naively" is the correct stance.

4.1 Approach A: loader preload (mfsBSD pattern)

Add to cdroot's /boot/loader.conf:

mfs_load="YES"
mfs_name="/rootfs.uzip"
mfs_type="mfs_root"

The FreeBSD loader reads /rootfs.uzip from the cd9660 boot media at loader time, before kernel boot. Loads the entire file into RAM. Kernel sees md0 already configured at boot — backed by RAM, not by a vnode on cd9660.

After loader hands off to kernel: cd9660 mounts as initial root (for kernel binary access), but nothing on cd9660 is held busy by a vnode-backed mdconfig (because md0 came from the loader, not from a runtime mdconfig -t vnode -f /rootfs.uzip).

cd9660 is the rescue environment. It already contains /rescue/sh, /sbin/gunion, /sbin/mdconfig, /sbin/reboot, /sbin/kenv — everything /init.sh needs to run gunion-create and reroot. We do NOT build or load a separate "rescue.img" memdisk. The cd9660 IS the rescue env.

Flow:

# /init.sh on cd9660 (Option D shebang as PID 1):

# /dev/md0.uzip already exists, RAM-backed (loader preloaded rootfs.uzip)

# Layer gunion on top for writable upper
gunion create /dev/md0.uzip       # /dev/gunion0

# Hand kernel the new root and trigger reroot
kenv vfs.root.mountfrom="ufs:/dev/gunion0"
exec /sbin/reboot -r              # process is replaced; kernel reroots

After reboot -r: kern_reroot() calls vfs_unmountall() which unmounts cd9660 cleanly (no busy vnodes). vfs_mountroot() mounts /dev/gunion0 as the new kernel root. /sbin/init runs from the gunion-backed UFS. From this point the kernel root and userspace root are the same view; firmware loading works natively.

Pros: standard FreeBSD pattern (mfsBSD has used it for decades). Loader-handled, simple init.sh, no separate rescue.img to build or maintain.

Cons: loader has to slurp ~400MB into RAM at boot — slower boot by 5-15 seconds depending on read speed (USB stick, optical, network). Loader has size limits in some BIOSes/UEFI firmwares; needs verification on the target hardware.

4.2 Approach B: in-init.sh memdisk copy

Same end state, done in init.sh instead of loader:

# Inside /init.sh, before any other md/gunion setup:

# 1. Create a malloc-backed memdisk sized for the rootfs
mdconfig -a -t malloc -s 400m -u 1   # /dev/md1, RAM-backed, empty

# 2. Copy rootfs.uzip into it
dd if=/rootfs.uzip of=/dev/md1 bs=1m

# 3. Detach the vnode-backed md (if we created one)
mdconfig -d -u 0 2>/dev/null    # no-op if we never created md0

# 4. /dev/md1.uzip is now RAM-backed; cd9660 is no longer needed
gunion create /dev/md1.uzip      # /dev/gunion0

# 5. Set reroot target and trigger
kenv vfs.root.mountfrom="ufs:/dev/gunion0"
reboot -r

Pros: doesn't depend on loader behavior. Easier to debug (it's a shell script, can be poked at interactively).

Cons: dd-copy of ~400MB takes 1-3 seconds on RAM. cd9660 stays mounted for the duration of the copy. Init.sh becomes more complex and order-sensitive.

4.3 Recommendation

Try Approach A first — it's the canonical pattern, less code, and if the loader supports the file size we get a clean boot. Fall back to B if loader-preload fails on real hardware.

4.4 Approach C: separate rescue.img + dual-memdisk (mfsBSD pattern)

The mfsBSD-canonical alternative: build a tiny rescue.img (~30MB UFS with just init + tools) and have the loader preload it alongside rootfs.uzip. Kernel boots from rescue.img (RAM-backed memdisk); rescue's init.sh sets up gunion over the second RAM-backed memdisk; reroots to gunion-backed UFS. cd9660 may not even be mounted at runtime.

# cdroot/boot/loader.conf:
mfs_load="YES"
mfs_name="/rescue.img"
mfs_type="mfs_root"

# Second memdisk (rootfs payload) loaded as a separate kernel preload:
rootfs_load="YES"
rootfs_name="/rootfs.uzip"
rootfs_type="md_image"

# /init.sh inside rescue.img:
gunion create /dev/md1.uzip          # md1 is the rootfs payload
kenv vfs.root.mountfrom="ufs:/dev/gunion0"
exec /sbin/reboot -r

What this pattern fixes (for mfsBSD's use cases)

Why we're not adopting this

None of the scenarios where miniroot earns its complexity apply to our livecd today:

• We don't PXE-boot.
• We don't eject media mid-boot.
• We ship a single rootfs payload.
• We're not diskless.
• The cd9660 already provides offline-rescue behavior — its /rescue/sh + tools are reachable any time the cd9660 is mounted, which is "until reroot." Anything that fails before reroot lands the user in cd9660's rescue shell.

Cost of adopting it anyway: an entire second build pipeline (rescue.img generation, separate Makefile target, separate set of binaries to keep version-aligned with the rootfs), debug surface (two stages of init), and the mfsBSD-canonical pattern's quirks (loader configuration of multiple memdisks, kernel preload type for the secondary md, etc.). All for "the same outcome we already get from cd9660."

Verdict: keep this pattern documented as the right choice for mfsBSD-shape use cases (PXE, ejectable media, multi-payload, diskless rescue). If freebsd-launchd ever grows into one of those use cases — most plausibly an installer that wants to eject the USB stick after install completes — this is the right architecture to revisit. For the current livecd, Approach A (cd9660 as rescue + loader-preloaded rootfs) is correct and simpler.

5. Why gunion specifically (vs. unionfs, vs. tmpfs root)

gunion gives us the cleanest "writable root" semantic — no thinking about which subdirs are writable, no skeleton-population dance, no per-iface tmpfs-mount choreography. Everything is writable; writes go to RAM via gunion's upper layer; reads come from the RO uzip via gunion's lower.

Trade-off vs. RO UFS + tmpfs subdirs: gunion has a known performance overhead (block-level overlay isn't free) and write coalescing/syncing semantics that have caused issues historically. We moved off gunion to unionfs in freebsd-livecd-unionfs partly because of these.

For this experimental architecture, gunion is the right pick because it's the only writable-everywhere overlay that can be a reroot target. The performance trade-off is worth re-evaluating now that we have the rootfs entirely in RAM (less I/O contention than gunion-over-cd9660-uzip).

6. Phased delivery

7. Risks

8. Comparison with sibling livecd projects

9. Open questions

10. References

• FreeBSD kern_reroot: sys/kern/kern_shutdown.c in the FreeBSD source tree.
• FreeBSD vfs_mountroot: sys/kern/vfs_mountroot.c.
• gunion(8): man 8 gunion.
• mdconfig(8): man 8 mdconfig.
• reboot(8) (specifically the -r flag): man 8 reboot.
• mfsBSD source (canonical mfs-loaded-from-loader pattern): github.com/mmatuska/mfsbsd.
• Sibling livecd projects:
  • freebsd-livecd-gunion (the parent — gunion + chroot, working today)
  • freebsd-livecd-unionfs (unionfs + chroot)
  • freebsd-launchd (unionfs + Option D chroot, also has the symlink-trick fix)
• Companion plans: freebsd-livecd, SquashFS on FreeBSD research.