Building a modern, writable, tiny live ISO for FreeBSD

Squashfs in the FreeBSD kernel — what exists, what's missing, what to actually build.

1. State of squashfs on FreeBSD

1.1 In-tree: nothing

A grep for squashfs across sys/, sbin/, usr.sbin/, contrib/ in the current source tree returns zero hits. There is no module skeleton, no GEOM provider, no VFS implementation. There is no mount_squashfs mount helper, and makefs(8) doesn't know about the format.

1.2 GSoC 2023 — the closest existing kernel work stalled

• Project: "Port SquashFuse To The FreeBSD Kernel"
• Student: Raghav Sharma (raghav@FreeBSD.org), mentored by Chuck Tuffli
• Code: github.com/Mashijams/freebsd-src — branch gsoc/squashfs
• Status reports: Q2 2023, Q3 2023
• What works: read-only mount, vop_lookup / vop_readdir (directories), vop_read / vop_strategy (files), vop_readlink (symlinks). Compression: zlib, lzo2, zstd. Targets FreeBSD 13.2-RELEASE.
• What's missing / unfinished: kernel xattr hooks pending, never merged, no Phabricator review, no quarterly mention from Q4 2023 → Q4 2025.
• Practical takeaway: the code is your starting point if you want a real kernel module — but you'd be reviving an unmaintained branch, rebasing onto current main, fixing whatever's broken under the new VFS, and adding tests. Estimate: weeks of focused work, not days.

1.3 squashfuse (FUSE) — works, but not for early boot limited

• Port: filesystems/fusefs-squashfuse (was sysutils/, recategorized 2024-11-06).
• Upstream: github.com/vasi/squashfuse.
• Provides squashfuse + squashfuse_ll, depends on FUSE 3, lz4, lzo2, zstd.
• Why it's awkward as a boot root: FUSE needs userspace, so the kernel can't mount squashfs as the initial root. You'd need an mfs_root with enough userspace to start squashfuse, then pivot — a lot more moving parts than just using a uzip image.

2. Your four real architecture options

3. In-tree vs. out-of-tree

3.1 An out-of-tree filesystem module is fully supported

FreeBSD's loader searches both /boot/kernel and /boot/modules by default — see sys/kern/kern_linker.c and stand/common/module.c:

static char linker_path[MAXPATHLEN] = "/boot/kernel;/boot/modules";
TUNABLE_STR("module_path", linker_path, sizeof(linker_path));

/boot/modules is the conventional drop-zone for third-party modules — it's exactly what nvidia, vendor wifi drivers, and various ports use.

3.2 A filesystem is a regular kernel module

The macro VFS_SET in sys/sys/mount.h is the registration hook, and it's part of the public KBI shipped in /usr/include/sys/. Every in-tree FS uses the same pattern; cd9660, fusefs, autofs, ext2fs, tarfs all build as both static and loadable.

VFS_SET(squashfs_vfsops, squashfs, VFCF_READONLY);
MODULE_VERSION(squashfs, 1);

3.3 Out-of-tree skeleton (the nvidia pattern)

An out-of-tree port-ready Makefile looks like this, leveraging /usr/share/mk/bsd.kmod.mk:

# Makefile in your repo
KMOD=    squashfs
SRCS=    squashfs_vfsops.c squashfs_vnops.c squashfs_io.c \
         squashfs_decompress.c \
         vnode_if.h opt_kstack_pages.h
KMODDIR= /boot/modules        # default is /boot/kernel; set to modules

.include <bsd.kmod.mk>

Build it against installed headers (no source tree required) by setting KERNBUILDDIR=/usr/include/sys. The ports framework (USES=kmod) handles all of this for you.

3.4 KBI caveat

FreeBSD's KBI is stable within a STABLE branch (e.g. 14.x), but not across major versions. You'd ship one binary per major release, or a source port that rebuilds locally. This is the standard third-party module trade-off.

3.5 Verdict

Out-of-tree is the right answer for an experiment, a third-party project, or a "live distro toolkit". You only need to upstream into sys/fs/squashfs if you want it loaded by the loader as part of a release build, or if you want it in /usr/include/sys for other consumers. Even then, you can develop entirely as a port and merge later.

4. Rescue and userland tooling

4.1 Mount helper

Each in-kernel FS has a corresponding sbin/mount_* helper that just calls nmount(2) with the right fstype. Look at sbin/mount_cd9660/ as the template — it's ~250 lines. Your mount_squashfs would be similar.

4.2 Rescue integration

rescue/rescue/Makefile already statically links mount_cd9660, mount_msdosfs, mount_nullfs, mount_unionfs, mount_udf, mount_nfs. Adding mount_squashfs is one line, but only matters if squashfs is in tree. For an out-of-tree port, ship the binary in /usr/local/sbin/.

4.3 mksquashfs / unsquashfs

Already packaged: sysutils/squashfs-tools. You don't have to build these yourself — your live-ISO build script depends on the port.

4.4 makefs(8) extension optional

If you want makefs -t squashfs to work natively (so the release build can produce squashfs images without external tools), add a backend in usr.sbin/makefs/ alongside cd9660.c. This is purely cosmetic — mksquashfs from the port already works.

5. Modern live ISO architecture

5.1 What's wrong with the existing FreeBSD installer ISO?

The release ISO (release/amd64/mkisoimages.sh) is a plain cd9660 image with /etc/fstab set to mount root ro. rc.initdiskless creates tmpfs overlays for /etc, /var, /tmp from cpio templates in /conf/. It works, but it's not compressed and the "writable" parts are limited to a few directories. There's no whole-rootfs writability.

5.2 The Linux pattern, translated to FreeBSD

5.3 Concrete boot flow

1. Loader (BIOS or EFI) reads cd9660; loads kernel + mfs_root image (small UFS, ~5–10 MB).
2. Kernel boots with mfs_root as /. Init runs a tiny /sbin/init shell script.
3. Script: locates the compressed image on the optical/USB media (or in cd9660), attaches it via mdconfig -t vnode -f /cdrom/rootfs.uzip, mounts the resulting /dev/mdN.uzip read-only at /newroot.
4. Script: creates a swap-backed memdisk for the writable upper (mdconfig -t swap -s 2g), then either
   • gunion create md0.uzip md1 → put a UFS on the resulting union device, mount at /newroot; or
   • mount tmpfs on /newroot.upper, mount unionfs-fuse with lower=uzip, upper=tmpfs.
5. Set vfs.root.mountfrom to the new device, reboot -r.
6. Kernel re-mounts root from the union device, re-execs /sbin/init from the live system. You're now running with a fully writable rootfs.

5.4 The unionfs trap

From mount_unionfs(8) in the current tree:
"THIS FILE SYSTEM TYPE IS NOT YET FULLY SUPPORTED (READ: IT DOESN'T WORK) AND USING IT MAY, IN FACT, DESTROY DATA ON YOUR SYSTEM."

Don't use it. NomadBSD switched to unionfs-fuse after deadlocks during reboot -r. Your two real overlay choices are gunion(8) (block-level, in-tree as of 14) or unionfs-fuse (file-level, port).

5.5 What existing FreeBSD-derived live distros do

Nobody uses squashfs. Every active project uses mkuzip.

6. reboot -r (reroot)

The FreeBSD analog of Linux's switch_root. Implemented in sbin/init/init.c (the userland half) and sys/kern/kern_shutdown.c (kern_reroot()). Restricted to PID 1.

What it does, mechanically:

1. Init copies itself into a tmpfs at /dev/reroot and re-execs from there.
2. The temp init issues reboot(RB_REROOT).
3. Kernel kills all other processes, unmounts everything except /dev, calls vfs_mountroot() with the new vfs.root.mountfrom.
4. Kernel re-execs /sbin/init from the new root.

What this means for you:

• You can absolutely "boot tiny mfs → mount big root → pivot". This is the canonical FreeBSD live pattern.
• Memory disks do survive a reroot (since the kernel hasn't been reset). Useful for the upper overlay tmpfs.
• It's not a writable-overlay creation mechanism by itself — it just switches the root. You build the union/overlay before calling reboot -r.

7. Recommended path forward

7.1 If your goal is a tiny modern writable live ISO ASAP do this

Don't write a kernel module. Build with what's already in the tree:

1. Stage your rootfs as UFS (use makefs -t ffs).
2. Compress with mkuzip -j16 -d -s 65536 rootfs.ufs (zstd, 64K blocks). Expect ~40-50% of original size for a typical install.
3. Build a tiny mfs_root (~5 MB UFS) containing /sbin/init, mdconfig, mount, gunion, your pivot script, and a bare /dev.
4. Pack everything into a cd9660 with makefs -t cd9660 -o rockridge,bootimage=....
5. Pivot script: attach uzip → create gunion with swap-backed upper → put UFS on union → reboot -r.
6. For installation, your installer reroots into the on-disk target.

Reference implementations to read first: helloSystem ISO build, NomadBSD build script.

7.2 If you specifically need squashfs format heavier

1. Fork the GSoC 2023 branch (Mashijams/freebsd-src).
2. Rebase onto current main; convert to a standalone repo with the layout squashfs/{Makefile, squashfs_vfsops.c, squashfs_vnops.c, ...}.
3. Add a mount_squashfs userland binary under squashfs/mount_squashfs/.
4. Add a ports skeleton (filesystems/squashfs-kmod + USES=kmod) that builds against installed headers and installs to /boot/modules/squashfs.ko.
5. Write a small test harness — at minimum, mount and read a few reference squashfs images produced by mksquashfs at different compression levels.
6. Then plug the resulting kmod into the live ISO build above, replacing mkuzip+mdconfig with mksquashfs+mount_squashfs.

You can do this entirely outside the FreeBSD tree. Upstream later if it's healthy.

7.3 If you want squashfs now without writing kernel code workable

Use fusefs-squashfuse from ports as the read-only layer. Boot scaffolding looks like (1) tiny mfs_root with fusefs.ko preloaded, squashfuse in /sbin, (2) script attaches the squashfs image, squashfuse rootfs.sqsh /newroot, (3) gunion-style overlay on top (note: gunion is block-level so it sits under, not above, FUSE — for FUSE you'd need unionfs-fuse instead), (4) reboot -r. Slower than uzip and an extra moving part, but no kernel code.

8. What gets touched, where

9. References

FreeBSD source (this tree)

• sys/fs/tarfs/ — modern in-tree precedent for a compressed read-only filesystem (zstd-tar). Read this first if you want to know what a clean modern FS module looks like.
• sys/fs/cd9660/, sys/modules/cd9660/Makefile — minimal example.
• sys/sys/mount.h — VFS_SET macro definition.
• sys/geom/uzip/ — geom_uzip implementation (CLOOP-format, zlib/lzma/zstd).
• sys/dev/md/md.c — memory disk driver, MD_ROOT mfs_root mechanism (lines 173-198, 2057-2062).
• sys/kern/kern_shutdown.c — kern_reroot() implementation.
• sbin/init/init.c — userland half of reboot -r.
• sbin/mount_cd9660/ — mount helper template.
• libexec/rc/rc.initdiskless — current FreeBSD writable-overlay-on-readonly-root mechanism (limited to specific dirs).
• release/amd64/mkisoimages.sh, release/amd64/make-memstick.sh — current ISO build.
• share/mk/bsd.kmod.mk — build infrastructure for kernel modules; supports out-of-tree via KERNBUILDDIR.
• tools/tools/nanobsd/nanobsd.sh, defaults.sh — read-only-root reference (root_rw_mount="NO").

External — squashfs-on-FreeBSD

• GSoC 2023 status reports: Q2, Q3.
• GSoC code branches: Mashijams/freebsd-src, sleipbyte/freebsd-src.
• squashfuse upstream: vasi/squashfuse.
• FreshPorts — filesystems/fusefs-squashfuse.
• Forum discussion — mksquashfs/unsquashfs/squashfuse on FreeBSD.
• squashfs format reference — kernel.org docs.

External — FreeBSD live-ISO building

• helloSystem boot architecture — hellosystem.github.io/docs/developer/boot.html.
• helloSystem build script — helloSystem/ISO/build.sh.
• NomadBSD build — nomadbsd/NomadBSD/build.
• GhostBSD build — ghostbsd/ghostbsd-build.
• mfsBSD paper — people.freebsd.org/~mm/mfsbsd.
• geom_rowr (helloSystem out-of-tree overlay GEOM) — hellosystem.github.io geom_rowr.
• helloSystem unionfs deadlock issue — helloSystem/ISO#326.
• 2025 unionfs writeup — tunbury.org/2025/09/17/freebsd-unionfs.

External — manuals & reviews

• geom_uzip(4)
• mkuzip(8)
• gunion(8) + Phab D32697
• reboot -r writeup + D2698 / D3693
• Linux livecd architecture (for comparison) — major.io article, Ubuntu LiveCDCustomization.

Generated 2026-05-04. Research synthesized from a deep search of /Users/jmaloney/freebsd-src (current main, last commit 045a9ef829fa) plus external research across FreeBSD status reports, GSoC archives, and the active FreeBSD-derived live-distro projects.