Why a FreeBSD live ISO won't boot under Ventoy / iPXE

The bootloader–kernel handoff wall, what Linux's initramfs really does about it, and what the NextBSD equivalent looks like. Background research for nextbsd#288.

1. The symptom

The NextBSD live ISO boots end-to-end from a real USB stick and in a VM, but #288 reports it failing under Ventoy and PXE/iPXE with a tell-tale console trail:

[init] media: rootfs.uzip-MISSING
mdconfig: realpath: No such file or directory
[init] uzip dev: md1.uzip-ABSENT
mount: /dev/md1.uzip: No such file or directory
[init] rofs lower: /rofs-EMPTY
[init] union assembled; ...
exec: /sbin/launchd: not found  ->  init died
panic: Going nowhere without my init!

Crucially, this is not the classic upstream "cd9660 mountroot error 19" panic. NextBSD's kernel root is ufs:/dev/md0 — the in-RAM mfsroot — which mounts fine on every transport. The failure is one layer up, in the userland /init script: it mounts cd9660:/dev/iso9660/NEXTBSD to reach a loose rootfs.uzip on the boot media, and that device never appears. The same trail can also show up on real hardware (observed on a Dell Wyse 5070 / Intel J4105 thin client) — see §9 for why the same symptom there usually has a different, easier cause.

2. The bootloader–kernel handoff wall

The central misconception is "the bootloader failed to produce /dev/cd9660." The bootloader does not produce that device at all. There are two separate device worlds with a hard wall between them:

   BOOTLOADER world                  | ExitBootServices |   KERNEL world
   (FreeBSD loader / GRUB / boot-132) |   -- handoff --  |   (FreeBSD kernel)
                                      |                  |
   reads files via FIRMWARE:         |                  |   builds /dev from SCRATCH
   * UEFI Block I/O protocol         |                  |   via its OWN native drivers:
   * BIOS INT 13h                    |                  |   * xhci/umass -> da0
   "disk0:" / currdev                |                  |   * ahci/ata   -> cd0/ada0
                                     ---> firmware dies  |   * geom_label -> /dev/iso9660/LABEL

When you boot via Ventoy or iPXE, the bootloader succeeds — firmware (Ventoy's INT 13h hook / iPXE's EFI Block I/O) is alive and serving the emulated disk, so the kernel and mfsroot load. Then ExitBootServices() fires, the firmware disk service is destroyed, and the kernel starts over with its own drivers talking to real silicon. /dev/cd9660 (really /dev/cd0 or /dev/iso9660/<LABEL>) is created here, by the kernel, only when geom_label tastes a provider carrying the ISO PVD magic CD001 at offset 0x8000 (sys/geom/label/g_label_iso9660.c).

Therefore a different bootloader cannot fix this. Every bootloader does the same thing: read via firmware, hand off, exit. None is present when the kernel builds /dev. (And Apple's boot-132 boots a Mach-O XNU kernel, not the FreeBSD ELF kernel NextBSD runs — it's both the wrong layer and the wrong kernel.)

3. Why it fails, per transport

The important nuance: Ventoy is not a firmware-wall problem. The USB stick is real hardware the kernel enumerates as da0; the ISO is merely a fragmented file on it that nothing reconstructs. iPXE/PXE/MEMDISK are the true firmware-wall cases (no real device exists).

4. The firmware limits (the hard constraints)

5. How Linux actually fixes it

Linux hits the same wall and "fixes" it by never asking the kernel to mount the boot device. Everything hard happens in the initramfs — a RAM image the bootloader hands over before ExitBootServices, so it survives the wall. The kernel's mountroot stays dumb; the initramfs is full userland and does the work, split by transport:

5.1 Ventoy — reconstruct the ISO from the real stick

The kernel sees the real USB stick (sdX); the ISO is just a fragmented file on it. The initramfs reconstructs it two ways:

1. Mount the data partition (exFAT) and loop-mount the ISO file. The filesystem resolves fragmentation transparently — loop-mounting a file never cares whether it's contiguous. This is what Debian/Ubuntu's iso-scan/findiso hooks do. The enabler: Linux has had exFAT in-kernel since 5.4, and Ventoy's data partition is exFAT.
2. Device-mapper / vtoyboot. Inject a dm-linear target that remaps the ISO's on-disk byte segments into a contiguous virtual device (used mainly for persistence). This is the Linux analog of FreeBSD's geom_ventoy.

5.2 iPXE / PXE — fetch the root over the network

No real device exists, so the initramfs pulls the root over the network into RAM:

• HTTP squashfs — dracut root=live:http://… / livenet, or casper netboot=url url=http://…iso (downloads + loop-mounts). The common path. Root in RAM.
• NFS root — kernel mounts / over NFS.
• iSCSI + iBFT — the one case a real block device persists: a kernel iSCSI initiator reads the iBFT table iPXE left and reconnects the target.

Debunking "Linux mounts the ISO over iPXE." It does not. "Ubuntu boots fine over iPXE" means kernel + initrd loaded by iPXE, then the initrd fetches the squashfs/ISO over the network into RAM. The block device is never mounted by the running kernel. So "boot like Ubuntu" is the fetch-to-RAM pattern.

6. The NextBSD equivalent

Linux's whole trick — kernel mountroot stays dumb, the initramfs is smart — maps directly onto NextBSD, because NextBSD already has the initramfs: the mfsroot + /init (the md_image the loader preloads, then init_path=/init). Today /init does exactly one thing — wait for /dev/iso9660/NEXTBSD, mount cd9660, find rootfs.uzip — then assemble a tmpfs+unionfs overlay and sysctl vfs.pivot=/rofs into it.

The fix is to make /init an ordered probe, the same shape as Linux's iso-scan → livenet chain:

locate rootfs.uzip:
  1. LOCAL cd9660   -> CD/DVD, dd-USB, VM virtual CD   (wait on cd0/da*/iso9660-label)
  2. VENTOY         -> mount the data partition, loop-mount the .iso, cd9660 it
  3. NETWORK        -> dhclient + fetch the uzip/ISO over HTTP/NFS  (iPXE/PXE)
  else panic with a clear, specific message

Each strategy just has to end with a path to rootfs.uzip; the existing union + vfs.pivot + exec /sbin/launchd tail is unchanged.

6.1 Architecture parity

7. Component assessment

7.1 geom_ventoy — BSD-licensed, but bake-in doesn't self-trigger caveat

• License: SPDX-License-Identifier: BSD-2-Clause-FreeBSD, a documented derivative of FreeBSD's sys/geom/concat/g_concat.c (the pjd 2004–2005 copyright is retained). No GPL taint — the GPL parts are Ventoy's GRUB layer, never linked into the kernel. Safe to bake in and redistribute (as a NextBSD-owned patch/overlay, never an edit to the fork).
• KBI: a single ~1.2k-line file pinned to G_VERSION; a 15.x/ source dir already exists and g_ventoy_class is byte-identical to 14.x. Per-major refresh cost is small.
• The catch: the class needs Ventoy's segment map (the fragmented ISO's byte ranges). Ventoy delivers it — via an in-memory magic-guarded struct or hint.ventoy.* loader hints — only when it detects and injects its own geom_ventoy.ko anchor (it keys on /boot/kernel/geom_ventoy.ko in the ISO). A statically baked-in class with no .ko on the ISO gets neither the map nor the hints, so it tastes nothing. Making bake-in actually function requires Ventoy-side cooperation: ship a stub .ko anchor so Ventoy fills the map, or get Ventoy to emit hint.ventoy.* for NextBSD unconditionally.

7.2 exFAT — the clean Linux path is blocked on FreeBSD gap

The cleanest "do what Linux does" for Ventoy is mount the data partition + loop-mount the ISO file. FreeBSD base has no exFAT — support is the FUSE port fusefs-exfat (a module + userland), impractical in early /init. Options: port/bake a read-only exFAT reader (most work), or require the Ventoy data partition be a FreeBSD-mountable FS (UFS, or FAT32 for ISOs < 4 GB) and loop-mount that — zero kernel work, but a user-formatting requirement.

7.3 iSCSI + iBFT — the one true runtime block device, but a kld policy clash

iPXE sanboot iscsi: writes an iBFT table; FreeBSD's isboot reads it and re-attaches the target with the native initiator, so the kernel genuinely sees a persistent device at runtime — the only non-RAM, in-kernel iPXE path. But isboot is a port/module (net/isboot-kmod). NextBSD uses kexts, not kldload, and prefers baked-into-kernel; a baked-in iSCSI-root would have to compile the initiator + iBFT path into the NEXTBSD kernel. Heavier than the network-fetch path and target-infrastructure-dependent.

8. Implementation plan

All of this is module-free — /init + loader.conf + baked-in-kernel bits. No klds, no bootloader swap, no edit to the freebsd-src fork (loader/userland changes ride the overlay-patch / build mechanism the same way kernel patches do).

Phase 0 — Harden local-media discovery cheap, do first

• Bump the root-mount wait and poll it finer; wait on cd0/1, da0..N, and iso9660/*, not just two node names. The timeout is an upper bound with early-exit — fast media still proceeds in ~1–2 s (see §9).
• Probe-mount each candidate and accept only if /media/rootfs.uzip exists (so a whole-device dd'd ISO is found as da0).
• Fixes burned CD/DVD, dd'd USB, VM — the majority of users — and very likely the real-hardware (Wyse 5070) case. Pure /init + loader.conf.

Phase 1 — Network fetch for iPXE/PXE the "boot like Ubuntu" path

• When strategies 1–2 find nothing, /init brings up the NIC (dhclient), then fetch -o /tmp/rootfs.uzip ${live_url}/rootfs.uzip into tmpfs and mdconfig -f it — literally casper netboot=url, FreeBSD-side.
• URL comes from a kenv/loader var (e.g. live.fetch_url), set in the iPXE script's imgargs / DHCP option.
• Needs the NIC driver baked into the kernel (compiled in, not a kext/kld) and dhclient+fetch added to the mfsroot tool set. Root is RAM-resident here — unavoidable (no device exists), and exactly what Ubuntu does.

Phase 2 — Ventoy spike first

Pick among: (2a) bake a read-only exFAT reader so /init can loop-mount the ISO (most Linux-faithful, most work); (2b) bake in geom_ventoy + ship a stub anchor and confirm Ventoy fills the map (§7.1); (2c) require a FreeBSD-mountable Ventoy data partition (UFS/FAT32) + loop-mount in /init (zero kernel work, documented stopgap).

PR breakdown

1. PR 1 (Phase 0) — /init ordered-probe scaffold + local-media hardening + timeout. Self-contained. (nextbsd-work)
2. PR 2 (Phase 1) — network-fetch strategy + bake NIC drivers into config/NEXTBSD + add dhclient/fetch to mfsroot. (nextbsd-work + nextbsd-kernel)
3. Spike → PR 3 (Phase 2) — commit to 2a/2b/2c with real data.

9. Diagnosing real hardware — same symptom, different cause

On a real machine (e.g. the Dell Wyse 5070 / J4105) the same rootfs.uzip-MISSING line usually has a different, easier cause than Ventoy:

The disambiguating test (boot the ISO with -s — /init drops to a miniroot shell; wait ~10–15 s, then):

ls /dev/da* /dev/cd* /dev/iso9660/
mount -t cd9660 -o ro /dev/da0 /media   # try da1/cd0 too
ls /media                               # rootfs.uzip there?
# if present: geom disk list / camcontrol devlist

On a real dd'd stick you can always eventually mount it by hand — that proves timing/discovery (Phase 0), not the firmware wall. On Ventoy you never can. That single manual mount tells you which problem you're solving before writing any code.

On NextBSD's mountroot: the kernel roots from ufs:/dev/md0 (the mfsroot), present instantly — so vfs.mountroot.timeout barely applies. The wait that matters is the userland /init poll loop, which early-exits the moment the device node exists. Raising it to 10 s costs nothing on fast media; the full 10 s is only spent when no local cd9660 is coming — which is precisely the signal to fall through to the next strategy.

10. References

NextBSD / project

• Issue — nextbsd#288: ISO won't boot via Ventoy/PXE
• Live-ISO build + mfsroot /init — nextbsd-work/build.sh (the /init heredoc, zz-live.conf, makefs mfsroot step)
• The in-kernel pivot primitive — nextbsd-kernel/src-overlay/sys/kern/vfs_pivot.c
• Companion site research — SquashFS on FreeBSD, FreeBSD Live ISO plan

FreeBSD source

• sys/kern/vfs_mountroot.c — root_mount_timeout, the device-present poll, the mountroot prompt
• sys/geom/label/g_label_iso9660.c — PVD taste at 0x8000, the CD001 magic, iso9660/<LABEL>
• stand/common/boot.c — getrootmount / currdev → vfs.root.mountfrom
• md(4), diskless(8), isboot (iBFT)

Firmware & prior art

• UEFI 2.9A §7 — Boot Services / ExitBootServices
• Ventoy — geom_ventoy source (BSD-2-Clause), Ventoy#2775 (pfSense/FreeBSD mount error 19)
• vtoyboot (Linux dm mechanism), Ventoy memdisk mode
• dracut.cmdline(7) — livenet / root=live:, iPXE sanboot
• mfsBSD — the FreeBSD md-root / PXE precedent

Synthesized from the nextbsd#288 investigation — parallel research passes across freebsd-src (loader→kernel root handoff, geom_label, vfs_mountroot), Ventoy/vtoyboot and iPXE prior art, the UEFI/BIOS firmware limits, and the local NextBSD mfsroot/vfs_pivot build. Corrects two common assumptions: that a bootloader swap could help, and that Linux mounts the ISO device at runtime over iPXE. Background reference for the #288 fix plan.