NextBSD — UFS + ZFS image/install strategy (options survey)

The three constraints everything is judged against

Simple installer. Keep the current cpdup whole-disk clone — no FS-choice/partitioning wizard.
No divergence: what you run from an image must be what gets installed — never "works differently between img and installed."
Live ISO stays UFS. ZFS is a supported installed option, never the live ISO's filesystem.

What the survey finds: only two option-families satisfy all three cleanly — A (per-FS prebuilt images: iso/ufs-img/zfs-img) and its packaging variants E/F. B (installer picks the FS) breaks constraint 1 and arguably 2; C (one installer, any target) keeps the UX simple but moves ZFS correctness into installer code and weakens constraint 2. A gating fact for any ZFS path: OpenZFS must be compiled into the NO_MODULES kernel first. (Presented as options; you decide later.)

Recommended shape — three editions (naming: live / installer / disk)

Split the artifacts by intent — run it / install it / deploy it — and gate ZFS to the two non-live editions. That keeps ZFS off cheap USB flash (it never runs from removable media) and removes any live-vs-installed divergence (there's no live ZFS to diverge from). The live edition stays UFS.

Edition	Intent	Filesystem	Example filename (amd64, 20260717)
live	Run from media (has installer too)	UFS only	`NextBSD-amd64-20260717-live.iso.zip`
installer	Boots straight to the installer	installs UFS or ZFS	`NextBSD-amd64-20260717-installer.iso.zip` `NextBSD-amd64-20260717-installer.img.zip` (USB)
disk	Prebuilt system — `dd`/VM, no installer	UFS and ZFS	`NextBSD-amd64-20260717-disk-ufs.img.zip` `NextBSD-amd64-20260717-disk-zfs.img.zip`

Naming schemes A/B/D and the full rationale are in §5–§6; the USB/write-endurance reality that motivates gating ZFS is in §4. Decision still deferred — this is the recommended framing, not a commitment.

1Current state — the baseline

build.sh already produces two artifacts per arch, from one staged rootfs:

.img.zip — a GPT disk image, rw UFS root (makefs -t ffs … label=ROOTFS + ~1.5 GB headroom, + FAT ESP + freebsd-boot/gptboot). dd it to a disk or boot it directly. Never unions — it's a plain UFS system.
.iso.zip — the live ISO (UFS): compact makefs → mkuzip (zlib), a ~3 MB mfsroot/init, geom_uzip RO lower + tmpfs upper via unionfs → vfs.pivot → launchd.

The installer (nextbsd-installer/engine/do-install.sh) is a cpdup whole-disk clone: gpart a GPT (boot + ESP + freebsd-ufs), newfs -U -L NEXTBSD, mount, cpdup -i0 / $MNT (clones the running / through the VFS — "works the same on the union ISO and a plain image"), fix fstab/loader, install bootcode. No FS wizard, whole-disk only. The clone is filesystem-agnostic (cpdup is a file-level tree copy); only the target-prep (gpart+newfs) is UFS-specific.

So the base is already close to the lean Today = iso (live UFS) + ufs-img (rw-UFS, effectively already exists) + a simple FS-agnostic cpdup installer. Adding ZFS is mostly "produce a third artifact" — not "rebuild the installer."

2How a `zfs-img` is actually built (and a boot correction)

It parallels the existing UFS .img step — swap the root filesystem:

# build a ZFS-root pool image straight from the staged rootfs (FreeBSD 14+ makefs)
makefs -t zfs -o poolname=zroot -s <size> -o rootpath=/ \
       -o bootfs=zroot/ROOT/default [ -o fs='zroot/ROOT/default;…' … ] zroot.zfs $WORK/rootfs

# same GPT as the UFS img — only the root partition + BIOS boot block change
mkimg -s gpt -b .../pmbr \
      -p freebsd-boot:=.../gptzfsboot \        # gptzfsboot, not gptboot
      -p efi/efiboot0:=esp.img \                # the SAME FAT ESP as today
      -p freebsd-zfs/zroot:=zroot.zfs \          # freebsd-zfs, not freebsd-ufs
      -o NextBSD-<arch>-zfs.img

Correction: the loader reads ZFS directly — no separate boot FS needed Earlier scoping worried a ZFS root needs a readable non-ZFS boot filesystem. It doesn't: FreeBSD's loader reads ZFS directly — gptzfsboot (BIOS) / loader.efi (UEFI). The only "readable boot partition" is the FAT ESP that holds loader.efi — already in the UFS img. So a zfs-img is the identical freebsd-boot + ESP + <root> GPT, just freebsd-zfs + gptzfsboot. makefs -t zfs is FreeBSD's sanctioned way to build exactly this from a staged tree (it's how release(7) makes root-on-ZFS VM images).

The one gating prerequisite (all ZFS options) The NEXTBSD kernel is NO_MODULES, so OpenZFS must be compiled statically into the kernel before it can mount a ZFS root — a nextbsd-kernel change, not a build-script tweak. UNVERIFIED whether this has landed. This gates A, B, and C equally.

3The options

A · Prebuilt image per FS your lean · aligns with all 3

Ship three artifacts: iso (live UFS, unchanged), ufs-img (today's dd-able UFS disk image), zfs-img (new, per §2). User picks the artifact; each .img is the installed system — dd it, or run the cpdup installer live-from-ISO.

✅ Constraint 1 — installer needs zero FS logic (dd the img, or the existing UFS clone; ZFS is dd-only).
✅ Constraint 2 — the zfs-img you dd is the ZFS system; divergence is build-time only (two builds from one rootfs), not run-vs-install. (Inherent: the live iso is UFS/union while a zfs-img is ZFS — a presentation difference, forced by constraint 3, not a build/install divergence.)
✅ Constraint 3 — ZFS delivered only as an installed-system image.

Cost: three artifacts to build/test/publish per arch (was two); needs OpenZFS-in-kernel; pool size baked at build (dd to a bigger disk → a post-boot zpool online -e/autoexpand, vs UFS growfs).

B · Installer picks the FS at install time conflicts with 1 & (arguably) 2

One image set; the installer offers "Guided UFS" vs "Guided ZFS" like FreeBSD bsdinstall / GhostBSD — zpool create, the ROOT/default dataset tree, bootfs, ZFS fstab/loader, gptzfsboot.

❌ Constraint 1 — adds an FS-choice screen + a whole second target-prep branch in do-install.sh. This is precisely the wizard you want to avoid.
⚠ Constraint 2 — the ZFS system is synthesized by installer code at install time; nothing ZFS was run/verified live. That's the build-vs-install divergence.
✅ Constraint 3 — live stays UFS.

Trade: one artifact to publish, matches the most common BSD precedent — but ZFS correctness lives in installer code, not a tested prebuilt image.

C · One installer, cpdup-clone to any target simple UX, weaker constraint 2

Keep the single flow; let target-prep make either a newfs'd UFS or a fresh zpool. The cpdup step is unchanged (it's already FS-agnostic). Only steps 1–2 (partition+FS) and the fstab/loader rewrite differ.

🟡 Constraint 1 — no user-facing wizard if the FS is chosen by a flag/artifact rather than a prompt, but there's a ZFS target-prep branch in the code.
⚠ Constraint 2 — same as B: the ZFS on-disk result is installer-built from a UFS-union live source.
✅ Constraint 3.

Does cpdup UFS→ZFS work? Mechanically yes — it's a file-level tree copy, FS-independent. Caveats: OpenZFS-in-kernel; more fstab/loader rewriting; datasets must be pre-created/mounted before the clone if you want a multi-dataset layout. Untested end-to-end on NextBSD.

Other strategies

C1 · cpdup from a prebuilt zfs-img template (A+C) — clone/zfs recv from a mounted, verified ZFS template instead of the UFS union. Regains A's "prebuilt ZFS is what you install" identity with one installer; more moving parts.
D · ZFS boot environments (bectl) — orthogonal to delivery; the payoff that justifies ZFS at all (snapshots, rollback, safe pkg upgrade). Adds no installer complexity beyond ZFS itself.
E · VM-image variants (.raw/.qcow2/.vhd) — the same ZFS pool image emitted in hypervisor formats via mkimg -f, near-zero extra cost. Pure packaging.
F · dd-only / separate download — publish zfs-img as an independent "dd this for ZFS root" download; the installer never touches ZFS. Maximally honors constraints 1 & 2; cost is "ZFS users dd, can't use the guided installer." (This is TrueNAS-style: ship the FS, no FS prompt.)

4The USB / write-endurance reality

The dimension the artifact-production view misses: what actually happens running off a cheap USB stick that can't take many writes

The wear question turns on whether the stick is the running medium or just a transfer vehicle to a real disk.

On a USB stick	Writes hit the stick?	Cheap-flash verdict
Live ISO (UFS, uzip + tmpfs)	Near-zero — base is read-only, all writes go to RAM (tmpfs/swap)	✅ Kind by design — the classic live-USB pattern
rw-UFS img run from USB	Yes — `/var`, logs, atime write continuously	🟡 Usable, but wears + slow random writes on cheap sticks
zfs-img run from USB	Heavily — COW + metadata + 5 s txg commits + ZIL	❌ Worst on cheap flash
Any img as install vehicle (dd → boot → install to internal disk)	Only read during install; writes go to the internal disk	✅ Fine (one-time)

Why ZFS specifically is rough on a cheap USB stick Two compounding problems: (1) write amplification — ZFS is copy-on-write + checksummed + metadata-heavy, so it writes far more than a simple FS and wears low-endurance flash fast; (2) the flush-lying problem — many cheap USB bridges ignore cache-flush/FUA, and ZFS's integrity guarantee depends on flush actually committing the txg, so a power-loss mid-commit can corrupt the pool. The very devices that wear fastest also undermine ZFS's "always consistent" promise.

Guidance this yields: run-from-USB → the live UFS ISO (read-only base + tmpfs = near-zero stick writes — and it's UFS, reinforcing "the ISO stays UFS"). ZFS belongs on installed internal disks (SSD/HDD with real wear-leveling), written once by a dd-able image. "ZFS running persistently off a cheap USB stick" is the anti-pattern — if truly wanted, use a proper USB SSD (not flash) with tuning, and accept the flush caveat.

5Delivery model — three editions (recommended framing)

Split the artifacts by intent: run it / install it / deploy it — and gate ZFS to the non-live editions

Edition	What it does	Filesystem	USB-safe?
live	Boots to a running system from the media (has the installer available too)	UFS only (uzip + tmpfs)	✅ near-zero writes
installer	Boots straight to the installer; installs to a target disk	installs UFS or ZFS (binary whole-disk toggle)	✅ read-only during install
disk (a.k.a. raw / installed)	A prebuilt disk image that is the installed system — `dd` to a disk or attach to a VM & boot; no installer	UFS and ZFS variants	🟡 install-vehicle only

Why this is the tidiest model — it dissolves two worries at once

ZFS lives only in installer + disk, never live. That single rule means:

Write-endurance solved by construction — ZFS never runs from a USB stick; it only lands on a real target disk. The run-from-USB path stays the live UFS ISO.
The "img≡installed divergence" worry dissolves — there is no live ZFS to diverge from an installed ZFS. You only ever get ZFS as an install of the same NextBSD (same files/packages, ZFS just the substrate). No divergence axis exists.

The installer stays "simple" as long as its FS choice is a binary whole-disk toggle ("Install UFS / Install ZFS"), not a bsdinstall-style partitioning wizard. And disk is the purest img≡installed: dd it and it is the system.

Naming options

Scheme	Run	Install	Prebuilt disk
A	`live`	`installer`	`raw`
B	`live`	`installer`	`installed`
C (clearest)	`live`	`installer`	`disk`
D	`live`	`install`	`image`

What the full filenames look like

Extending the current convention NextBSD-<arch>-<date>.<type>.zip to NextBSD-<arch>-<date>-<edition>[-<fs>].<format>.zip (example: amd64, 20260717), under scheme C:

Edition	Filesystem	Format	Filename
live	UFS	iso	`NextBSD-amd64-20260717-live.iso.zip`
installer	installs UFS/ZFS	iso	`NextBSD-amd64-20260717-installer.iso.zip`
installer (USB memstick)	installs UFS/ZFS	img	`NextBSD-amd64-20260717-installer.img.zip`
disk	UFS	img	`NextBSD-amd64-20260717-disk-ufs.img.zip`
disk	ZFS	img	`NextBSD-amd64-20260717-disk-zfs.img.zip`
disk (VM formats, optional)	ZFS	qcow2 / vhd / raw	`NextBSD-amd64-20260717-disk-zfs.qcow2.zip` …

Under scheme A the two disk images become …-raw-ufs.img.zip / …-raw-zfs.img.zip; under B, …-installed-ufs.img.zip / …-installed-zfs.img.zip. Arm64 emits the same set with arm64 in place of amd64. (Today's build produces just NextBSD-<arch>-<date>.iso.zip + .img.zip — this scheme renames & extends that.)

6Comparison

Option	1 · Installer simple	2 · img≡installed	3 · ISO stays UFS	ZFS delivery	Artifacts / arch
Current	✅ simplest	✅	✅	none	iso + img (2)
A · per-FS prebuilt lean	✅ (dd / unchanged)	✅ per artifact	✅	`makefs -t zfs` → dd-able `zfs-img`	iso + ufs-img + zfs-img (3)
B · installer FS choice	❌ wizard + branch	⚠ built at install	✅	installer `zpool create`	iso + img (2)
C · cpdup any target	🟡 no wizard, code branch	⚠ built at install	✅	installer `newfs` or `zpool`	iso + img (2)
C1 · cpdup from zfs-img	🟡 one installer + template	✅ template	✅	ship + clone a `zfs-img`	iso + img + zfs template (3)
E/F · VM variants / dd-only	✅ (installer ignores ZFS)	✅	✅	dd-able img (raw/qcow2/vhd)	iso + ufs-img + zfs-img (+formats)

Aligns with all three constraints: A (the lean) and E/F (its packaging variants). B conflicts with 1 and arguably 2. C keeps the UX simple but moves ZFS correctness into installer code and weakens 2. (Not a pick — the decision is yours to make later.)

7Precedent

makefs -t zfs is FreeBSD's sanctioned way to build a ZFS-root image from a staged tree (14+), reproducing the bsdinstall dataset layout — used by release(7) for root-on-ZFS VM images. Maps directly onto NextBSD's existing makefs+mkimg step.
The loader reads ZFS directly (gptzfsboot/loader.efi) — corrects the "needs a readable boot FS" assumption; the FAT ESP already is that partition.
FreeBSD bsdinstall = the Option-B model (guided Auto-UFS vs Auto-ZFS at install time — the wizard being avoided). GhostBSD wraps the same choice.
FreeBSD release engineering already ships multiple prebuilt image variants per release (install media + dd-able VM images in raw/qcow2/vhd/vmdk, in both UFS and ZFS flavors) — the prebuilt-image-per-FS precedent (A/E), distinct from installer-choice.
TrueNAS = "ZFS-first, no FS wizard" — reinforces that "pick the artifact, no FS choice in the installer" (F) is a shipped design, not an omission.

Where it stands The current base already gives you iso(live-UFS) + ufs-img + a simple FS-agnostic installer; the zfs-img is a third build artifact (makefs -t zfs, same GPT + gptzfsboot), gated only on OpenZFS-in-kernel. Options A and E/F reach the iso/ufs-img/zfs-img future without touching the installer or diverging img-from-installed. B and C get ZFS with one image but at the cost of installer complexity and/or build-vs-install divergence. Nothing here is decided — this is the menu to weigh when you're ready.