A single-PR plan to move NextBSD's image build off the vmactions/freebsd-vm qemu VM (~17 min) and onto a native Linux runner that cross-compiles the entire Apple/Darwin userland for FreeBSD with Clang 19 — zero FreeBSD packages installed, nothing built in a chroot, the bootable UFS image assembled on Linux, and qemu used only to boot-test. The approach is not speculative: four sibling repos (nextbsd-kernel-toolchain, nextbsd-kernel, nextbsd-kernel-modules, nextbsd-freebsd-compat) already cross-build the FreeBSD base, kernel, and modules on ubuntu-24.04 with this exact pattern.
NextBSD today assembles its bootable image by running build.sh inside a vmactions/freebsd-vm qemu VM (~17 min). The two genuine blockers are (a) the in-chroot pkg bootstrap of cmake/ninja/llvm19 (build.sh:195-340) and (b) the two in-chroot cmake/ninja builds — libdispatch (build.sh:874) and swift-foundation-icu (build.sh:1203) — both of which execute FreeBSD binaries. The remaining 30 components already use bmake (make -C src/<comp>) and need only a cross wrapper, not a port.
The target end-state is not speculative — it is already proven in production by four sibling repos. nextbsd-kernel-toolchain, nextbsd-kernel, nextbsd-kernel-modules, and nextbsd-freebsd-compat *already* cross-build the FreeBSD base, kernel, and modules on a native ubuntu-24.04 runner using Clang 19 cross-compilation, zero FreeBSD packages, no chroot, no emulation. Critically, nextbsd-freebsd-compat already demonstrates the exact "world" pattern this PR needs: per-directory make -C <dir> wrapped in make.py buildenv with a Linux-safe install -N -U -M METALOG, producing a from-source FreeBSD sysroot. NextBSD already *consumes* that sysroot artifact (nextbsd-base-<arch>.tar.gz, extracted at build.sh:95).
So the work decomposes into:
buildenv + curated-install pattern with path edits. Low/moderate risk — mechanically proven.nextbsd-kernel-modules out-of-tree-kmod pattern verbatim. Low risk — it is a vanilla bsd.kmod.mk module.buildenv wrapper plus a one-time cc→$(CC) recipe sed (mechanical, gated by a static ELF check).MIGCC split — wiring, not invention.ubuntu-latest + qemu + OVMF (it was never vmactions).Why one PR. Every dependency is already built and published by the sibling chain; this PR *consumes* those artifacts and changes only (1) the nextbsd workflow, (2) build.sh's front-half/tail, (3) a new build-cross.sh driver, (4) two CMake toolchain files, and (5) tiny edits to two CMakeLists. Nothing requires bootstrapping new infrastructure — the toolchain image, the sysroot artifact, the ccache wiring, the dispatch cascade, and the boot-test harness all exist. The boot-test contract (~80 expect markers) is a byte-identical oracle: the same harness that gates the vmactions image today gates the Linux-assembled image, so "green boot-test" *is* the definition of parity. Failures localize to the exact component being built, so the branch can be grown in dependency order with each step independently verifiable.
The four sibling repos form a cascade releng/15.0 → toolchain → {kernel → modules, compat} → nextbsd. We reuse the following verbatim or near-verbatim:
| Asset | Source repo | What we reuse | ||
|---|---|---|---|---|
| Toolchain container image | nextbsd-kernel-toolchain | ghcr.io/nextbsd-redux/nextbsd-kernel-toolchain:<arch>-<tag> — bakes clang-19/lld-19 (apt.llvm.org), llvm binutils, bmake, full /usr/src (releng/15.0 incl. share/mk + tools/build/make.py), ccache, and ENV MAKEOBJDIRPREFIX=/usr/obj, CROSS_BINDIR=/usr/lib/llvm-19/bin, CCACHE_CROSS_BINDIR=/opt/ccache-cross. We run the nextbsd build *inside this image* as a container: job. | ||
buildenv curated-install template | nextbsd-freebsd-compat/.github/workflows/build.yml + srclist.txt | The make.py … buildenv BUILDENV_SHELL=/tmp/curated.sh loop doing per-dir make -C $d obj/all/install with DESTDIR=/stage, METALOG, and INSTALL="install -N /usr/src/etc -U -M /stage/METALOG -D /stage". This is the exact wrapper our 30 bmake components need. | ||
| Sysroot bootstrap pair | compat | make.py … WITHOUT_TOOLCHAIN/TESTS/LIB32/MAN=yes toolchain then _includes, plus the bulk header stage cp -RLp …/tmp/usr/include/. /stage/usr/include/ (the -L dereferences the symlinked headers). | ||
| Base sysroot artifact | compat | nextbsd-base-<arch>.tar.gz is a structurally complete clang --sysroot: full /usr/include, crt objects (csu), libc_nonshared.a, libcompiler_rt.a/libgcc.a, libthr, libc++/libcxxrt, every base .so. We extract it and pass --sysroot=<dir>. | ||
| Out-of-tree kmod cross recipe | nextbsd-kernel-modules | make.py --cross-bindir=$CCACHE_CROSS_BINDIR TARGET=… NO_KERNEL=yes against a downloaded kernel-obj tree — the direct precedent for mach.ko. | ||
| ccache block | kernel/modules | env CCACHE_DIR=${{github.workspace}}/.ccache; actions/cache@v4 key ccache-<job>-<target>-${{github.run_id}} + restore-keys prefix; ccache -o max_size=NG; ccache -z; ccache -s. | ||
| Matrix + image-tag expression | toolchain/kernel/modules | matrix.include:[{target:amd64,target_arch:amd64},{target:arm64,target_arch:aarch64}] and the `:${{matrix.target}}-${{… && format('fbsd-{0}', …) | 'latest'}}` tag expr. | |
| Cross-repo artifact download | modules | actions/download-artifact@v4 with repository:, run-id:, github-token: ${{secrets.DISPATCH_TOKEN}} (all three required or it 404s). | ||
| Dispatch cascade | toolchain/kernel/compat | gh api repos/…/dispatches -f event_type=… -f client_payload[fbsd_sha]=…, gated github.ref_name=='main' && matrix.target=='amd64'. nextbsd already consumes base-updated. |
The key inheritance: compat proves you can cross-build FreeBSD *userland* directories on Linux into a /stage sysroot with correct ownership via METALOG. The 30 Apple bmake components are the same kind of bsd.lib.mk/bsd.prog.mk directories — so the pattern transfers directly.
repository_dispatch base-updated / push main / workflow_dispatch
│
▼
┌──────────────────────────────────────────────────────────────────┐
│ build (matrix: amd64, aarch64-gated) runs-on: ubuntu-24.04 │
│ container: ghcr.io/.../nextbsd-kernel-toolchain:${target}-<tag> │
│ │
│ 1. checkout │
│ 2. ccache restore + reset (10G) │
│ 3. download artifacts (cross-repo, by run-id + DISPATCH_TOKEN): │
│ nextbsd-base-<arch> → extract to $WORK/rootfs (sysroot) │
│ kernel-obj-<arch> → /usr/obj (for mach.ko KBI) │
│ nextbsd-modules-<arch> → modules-artifact/ (non-fatal) │
│ 4. build HOST tools (native x86_64-linux, NOT --target): │
│ migcom/mig, makefs, mkimg, kldxref, pwd_mkdb, cap_mkdb │
│ 5. make.py toolchain && _includes (stage cross sysroot) │
│ 6. build-cross.sh: 30 bmake comps + mach.ko via make.py buildenv │
│ 7. cross cmake: libdispatch + swift-foundation-icu │
│ 8. assemble image on Linux (METALOG → makefs/mkimg, offline db) │
│ 9. ccache -s; upload NextBSD-<arch>-<date>.img.zip │
└──────────────────────────────────────────────────────────────────┘
│ needs: build
▼
┌──────────────────────────────────────────────────────────────────┐
│ test (matrix amd64→qemu-x86_64+OVMF; arm64→qemu-aarch64+AAVMF) │
│ runs-on: ubuntu-latest (NOT a container — needs /dev/kvm) │
│ download artifact → tests/boot-test.sh (UNCHANGED ~80 markers) │
└──────────────────────────────────────────────────────────────────┘
│ needs: [build, test], if push main
▼
┌──────────────────────────────────────────────────────────────────┐
│ release (single non-matrix job: gather both arches, publish once)│
└──────────────────────────────────────────────────────────────────┘Cross mechanism. No --target/--sysroot is hand-passed to bmake. We drive every FreeBSD make -C through ./tools/build/make.py --cross-bindir=$CCACHE_CROSS_BINDIR TARGET=$T TARGET_ARCH=$TA buildenv, which sets XCC/XCXX/sysroot automatically from the staged cross-tools. The sysroot for per-dir builds is the buildenv-managed /usr/obj/usr/src/<machine>.<arch>/tmp *plus* the extracted compat base in $WORK/rootfs (which doubles as the incremental sysroot as components install headers the next consumes).
No packages, no chroot, no FreeBSD-binary execution at build time. clang/lld come from the image; cmake/ninja/byacc/flex come from apt (baked into the Dockerfile in a follow-up). Every chroot operation in build.sh is either deleted (pkg bootstrap, devfs, host-coreutils copy) or replaced with an offline host-tool invocation (kldxref, pwd_mkdb, cap_mkdb) or a static inspection (llvm-readelf for the old ldd/ldconfig -r self-tests).
amd64 first. compat is amd64-only today; nextbsd-base-arm64 does not yet exist. The aarch64 build leg is wired but gated off until the compat arm64 base ships. arm64 is then a *matrix add*, not new engineering (the toolchain files are arch-parameterized).
Build order is dependency-driven: libmach → migcom(host) → libdispatch → libxpc → ICU → CoreFoundation → the MIG daemons → cmd suites → bootstrap/ssh-bonjour, with mach.ko independent. All 30 bmake components use the same make.py buildenv wrapper + install -N -U -M METALOG; the table notes only per-component specifics.
| # | Component | Build system | Difficulty | Action |
|---|---|---|---|---|
| 1 | libmach (libsystem_kernel) | bmake | moderate | First in chain. buildenv cross; pre-create /usr/lib/system + /usr/include (not auto-mkdir'd). Install .defs (std_types/mach_types/machine_types) FIRST — all downstream MIG consumes them. LIBADD+=execinfo. |
| 2 | bootstrap_cmds (migcom + mig) | bmake | moderate | HOST tool — build native (no --target) into $WORK/hosttools/{libexec/migcom,bin/mig}; apt byacc+flex. Drop chroot mig -version; verify natively. |
| 3 | libdispatch | cmake | hard | See §5. Cross toolchain file, HAVE_MACH=ON, host migcom + cross MIGCC, post-install SONAME symlinks. |
| 4 | libxpc | bmake | moderate | buildenv; route SYSROOT=$WORK/rootfs. Do not pass CFLAGS/LDFLAGS on cmdline (bmake overrides +=, build.sh:1014 warns). Links libsystem_kernel. |
| 5 | swift-foundation-icu | cmake | hard | See §5. Cross toolchain file, host-decoded data blob, default -jN, libicucore aliases. Build before CoreFoundation. |
| 6 | libCoreFoundation | bmake | hard | buildenv; SHLIB_MAJOR=6; pre-mkdir /usr/include/CoreFoundation. -fblocks pervasive (CF headers carry block typedefs → every consumer needs -fblocks). Depends on ICU+libdispatch+libsystem_kernel. |
| 7 | launchd (liblaunch/src/support) | bmake | hard | 3 subdirs. Pre-gen 7 .defs (job/job_forward/job_reply/internal/helper/mach_exc/notify) via cross-MIGCC+host-migcom into $WORK/launchd-mig; pass MIGOUT=. launchctl links CF+ICU → after CF. |
| 8 | configd | bmake | hard | Gen config.defs (base id 20000) into $WORK/configd-mig. Ships config_wire.c reused by libSystemConfiguration. Drop host CLI self-tests or build them cross. |
| 9 | libSystemConfiguration | bmake | hard | After configd; MIGOUT=$CONFIGD_MIG, reuses configUser.c+config_wire.c. CF-typed -fblocks. |
| 10 | hwregd | bmake | moderate | Gen hwreg.defs into $WORK/hwreg-mig. hwregUser.c reused by libIOKit. |
| 11 | libIOKit | bmake | moderate | After hwregd; MIGOUT=$HWREG_MIG. CF wrapper, -fblocks. |
| 12 | Libnotify (lib + notifyd) | bmake | hard | Gen notify_ipc.defs+notify_old_ipc.defs into $WORK/libnotify-mig; symlink short names (subsystem notify_ipc, build.sh:1974). MIGOUT=. |
| 13 | syslog (asl/syslogd/aslmanager/util) | bmake | hard | 5 subdirs, no top Makefile. Gen asl_ipc.defs into $WORK/asl-mig; all four take MIGOUT=. Order: asl→syslogd→aslmanager→util. |
| 14 | IPConfiguration | bmake | hard | Gen ipconfig.defs into $WORK/ipcfg-mig. CF/SC-typed. After libSystemConfiguration. |
| 15 | KernelEventMonitor | bmake | moderate | No MIG. SC publisher. After libSystemConfiguration. |
| 16 | mDNSResponder (daemon + libdns_sd) | bmake | moderate | No MIG. Verify __APPLE__/mDNSPosix guards pick FreeBSD path under cross-clang. dns_sd.h consumed by ssh-bonjour. |
| 17 | DiskArbitration | bmake | moderate | No MIG. CF/SC, -fblocks. |
| 18 | hostnamed | bmake | moderate | No MIG. CF/SC+libnotify. Drop/convert host self-tests. |
| 19 | OpenPAM | bmake | moderate | CC?=cc so CC override works. -DHAVE_CONFIG_H (vendored config.h, no live configure). Builds libpam.so.6 overlay. SYSROOT= staged. |
| 20 | pam_modules | bmake | moderate | Uses ${CC}. 5 pam_*.so.6 overlays. Depends on OpenPAM headers. |
| 21 | file_cmds | bmake (GNU-flavor) | hard | 17 recipes hardcode cc/install — sed cc→$(CC), install→$(INSTALL). Links -lutil/-lsbuf. |
| 22 | shell_cmds | bmake (GNU-flavor) | hard | 41 hardcoded cc — same sed. /bin/sh load-bearing. |
| 23 | text_cmds | bmake (GNU-flavor) | hard | 34 hardcoded cc — same sed. Do NOT build the vendored jq subtree via its configure. |
| 24 | adv_cmds | bmake (GNU-flavor) | moderate | 7 hardcoded cc — same sed. Provides cap_mkdb (image-critical). |
| 25 | system_cmds | bmake (GNU-flavor) | moderate | 11 hardcoded cc — same sed. Provides pwd_mkdb (image-critical). |
| 26 | bootstrap | inline cc | trivial | Replace inline host cc (build.sh:760-799) with cross-clang --target/--sysroot, -lsystem_kernel -lpthread. |
| 27 | ssh-bonjour | inline cc | trivial | Single cross-clang compile (build.sh:2631), -ldns_sd. After mDNSResponder. |
| 28-32 | (the 4 cmd-suite tools + bootstrap server are counted above; OpenSSH below) | autotools | hard | OpenSSH (build.sh:2594): cross-configure --host=<arch>-unknown-freebsd15.0 CC=cross-clang --sysroot=, with a cached config.cache for tests that try to run target binaries (classic autotools cross gotcha). |
| — | mach.ko | bmake (bsd.kmod.mk) | hard | Vanilla out-of-tree kmod. make -C src/mach_kmod SYSDIR=/usr/src/sys KMODDIR=/boot/kernel KERNBUILDDIR=<kernel-obj>/sys/NEXTBSD inside make.py buildenv --cross-bindir=$CCACHE_CROSS_BINDIR. Highest-confidence hard piece — modules repo already proves this pattern. KERNBUILDDIR must come from the kernel-obj of the *matching fbsd_sha run* (KBI match). |
Linux-install requirement (all bmake components): every install uses install -N /usr/src/etc -U -M $WORK/rootfs/METALOG -D $WORK/rootfs — -N supplies the FreeBSD user/group DB (no wheel on Linux), -U is metalog mode (Linux can't chown). The chflags -R noschg cleanup is replaced with plain rm (no schg on Linux).
Both keep their upstream CMake builds (rewriting to bmake is enormous and pointless) and become native-Linux cross builds driven by a CMAKE_TOOLCHAIN_FILE pinned to FreeBSD. apt-install cmake ninja-build (bake into the Dockerfile). Shared toolchain file shape:
# cmake/freebsd-cross-toolchain.cmake
set(CMAKE_SYSTEM_NAME FreeBSD)
set(CMAKE_SYSTEM_PROCESSOR x86_64) # aarch64 for arm64
set(_triple x86_64-unknown-freebsd15.0) # aarch64-...-freebsd15.0
set(CMAKE_C_COMPILER /opt/ccache-cross/clang) # ccache-wrapped
set(CMAKE_CXX_COMPILER /opt/ccache-cross/clang++)
set(CMAKE_ASM_COMPILER /opt/ccache-cross/clang)
foreach(l C CXX ASM)
set(CMAKE_${l}_COMPILER_TARGET ${_triple})
endforeach()
add_link_options(-fuse-ld=lld)
set(CMAKE_SYSROOT $ENV{ROOTFS}) # = $WORK/rootfs (extracted base)
set(CMAKE_FIND_ROOT_PATH ${CMAKE_SYSROOT})
list(APPEND CMAKE_PROGRAM_PATH $ENV{HOSTTOOLS}/bin) # host mig
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) # mig is a HOST tool
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) # system_kernel under sysroot
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) # never host /usr/include
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)Pointing CMAKE_C/CXX_COMPILER at the ccache-wrapped /opt/ccache-cross/clang[++] (not the raw /usr/lib/llvm-19/bin) is what makes ICU's ~600 C++ TUs cache across runs.
The cross challenge reduces to four mechanical things; the CMakeLists' ~200 check_* probes already give the correct FreeBSD answers *if they compile-test against the sysroot*.
$WORK/hosttools (native clang, no --target; apt byacc+flex). Drop chroot mig -version.mach/mach.h umbrella until after libdispatch (build.sh:995) precisely to keep HAVE_MACH off. For HAVE_MACH=ON we must install mach/mach.h + mach type headers into $WORK/rootfs/usr/include before configuring libdispatch, so check_include_files(mach/mach.h) → HAVE_MACH=ON. If the full umbrella over-includes Apple-internal paths, install a curated mach.h exposing only the libsystem_kernel surface.protocol.defs: MIGCOM=$WORK/hosttools/libexec/migcom, MIGCC='/usr/lib/llvm-19/bin/clang --target=<triple> --sysroot=$WORK/rootfs' (so mig's internal cc -E resolves mach .defs imports from the sysroot). ``bash cmake -G Ninja -S src/libdispatch -B $WORK/libdispatch-build \ -DCMAKE_TOOLCHAIN_FILE=cmake/freebsd-cross-toolchain.cmake \ -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=lib/system \ -DINSTALL_DISPATCH_HEADERS_DIR=/usr/include/dispatch \ -DINSTALL_OS_HEADERS_DIR=/usr/include/os \ -DHAVE_MACH:BOOL=ON -DCMAKE_BUILD_TYPE=Release cmake --build $WORK/libdispatch-build DESTDIR=$WORK/rootfs cmake --install $WORK/libdispatch-build ``
Post-configure assert (hard gate): grep -q '#define HAVE_MACH 1' $WORK/libdispatch-build/config/config_ac.h — fail loudly if the sysroot's mach/mach.h was missing. Post-build assert: protocolServer.h + protocolUser.c exist non-empty (proves MIG fired).
Keep verbatim the SONAME symlink block (build.sh:894-918): libdispatch.so.0, libBlocksRuntime.so.0, libsystem_dispatch.so[.0], libsystem_blocks.so[.0] (libdispatch installs unversioned; FreeBSD ldconfig only indexes lib*.so.[0-9]+). BlocksRuntime cross-builds from src/BlocksRuntime (if(NOT APPLE) add_subdirectory) — plain C, no host tooling. find_library(system_kernel … PATHS /usr/lib/system NO_DEFAULT_PATH) (src/CMakeLists ~199): if NO_DEFAULT_PATH defeats sysroot rerooting, patch PATHS to ${CMAKE_SYSROOT}/usr/lib/system. Rebuild test_libdispatch/test_libdispatch_mach with cross-clang; replace chroot ldd/ldconfig -r with llvm-readelf -d DT_NEEDED checks.
The hard ICU problems are already engineered away in this fork. Grep confirms: no genrb/pkgdata/icupkg/genccode, no --with-cross-build, no try_run, no CMAKE_CROSSCOMPILING branch. CLDR data is pre-baked into 4 committed hex chunks. The OOM TU (icu_packaged_data.cpp) is already removed from all source lists; the .incbin/.S path bounds peak RSS to ~150 MB, so drop the stale -j2 cap and use default -jN.
The one cross fix: freebsd_decode_icu_chunks (icuSources/common/CMakeLists.txt:268) — a ~100-line host C program that decodes the hex chunks to a 40 MB binary blob the .incbin embeds. Under a cross toolchain file, add_executable would build it as a FreeBSD ELF that can't run on the Linux runner (Exec format error). Recommended (option iii, lowest-risk): decode the blob with native cc in a shell step before cmake, pass -DICU_PACKAGED_BIN_PATH=…, and delete the add_executable + add_custom_command + add_custom_target in common/CMakeLists.txt — keep only the configure_file of the .S template + target_sources of the generated .c + the uvernum.h symbol extraction.
Configure/build/install:
cc src/swift-foundation-icu/icuSources/common/<decoder>.c -o $WORK/decode && \
$WORK/decode … > $WORK/icu_packaged_main_data.bin # host pre-decode
cmake -G Ninja -S src/swift-foundation-icu -B $WORK/icu-build \
-DCMAKE_TOOLCHAIN_FILE=cmake/freebsd-cross-toolchain.cmake \
-DICU_PACKAGED_BIN_PATH=$WORK/icu_packaged_main_data.bin \
-DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=lib/system \
-DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release
cmake --build $WORK/icu-build # default -jN
DESTDIR=$WORK/rootfs cmake --install $WORK/icu-buildCMAKE_SYSTEM_NAME=FreeBSD fires the elseif(... FreeBSD) branch (line 99, no U_TIMEZONE). U_DISABLE_RENAMING=1 is forced unconditionally (line 61) so CF links ucol_open, not icu_74_ucol_open. cmake's plain-copy install needs no chown, so the Linux DESTDIR install Just Works (no -N/-U needed here). Recreate libicucore aliases (replacing build.sh:1225-1227): ln -sf lib_FoundationICU.so {libicucore.so,libicucore.so.74,lib_FoundationICU.so.74}. Build before CoreFoundation (CF #include <_foundation_unicode/uloc.h>).
ICU validation: llvm-readelf -s lib_FoundationICU.so | grep icudt74_dat shows a ~40 MB OBJECT (data embedded); llvm-nm -D | grep -c 'T ucol_open' == 1 and grep icu_74_ucol_open == 0; llvm-readelf -h shows OS/ABI FreeBSD + correct e_machine; DT_NEEDED lists FreeBSD libc++.so.1.
All image-time FreeBSD binaries are pure offline file/ELF/cdb processors — they read/write on-disk structures and never need a FreeBSD kernel. We cross-build them for the runner host (native x86_64-linux ELF, not --target=freebsd) and run them against the staged rootfs with no chroot.
| Step | build.sh | Linux replacement |
|---|---|---|
| pwd_mkdb (pwd.db/spwd.db) | 2675 | Offline pwd_mkdb -p -d $WORK/rootfs/etc …/master.passwd — same form build.sh:119 already uses today (proven). |
| cap_mkdb (login.conf.db) | 2676 | Offline cap_mkdb writing into rootfs/etc. Trivial cdb writer. |
| kldxref (linker.hints) | 645,653 | Offline host kldxref $WORK/rootfs/boot/kernel — parses each .ko MODULE_METADATA section (never executes). Already cross-used by nextbsd-kernel-modules. |
| ldconfig hints (×12) | 688,911,… | Drop (binaries bake -Wl,-rpath,/usr/lib/system) or one offline host ldconfig at the end. |
| ldd/ldconfig-r self-tests | 815-966,… | Static llvm-readelf -d DT_NEEDED checks (these never *ran* the binary — only inspected it). |
| libscan ELF closure | 2697-2715 | readelf→llvm-readelf. Diagnostic-only, never fatal. |
| makefs FFS root | 2718-2725 | makefs -t ffs -B little -o version=2,label=ROOTFS,softupdates=1 -b 1500m -F $WORK/METALOG rootfs.ufs $WORK/rootfs |
| makefs msdos ESP | 2731-2746 | makefs -t msdos -o fat_type=32 … (FAT has no uid/gid → no metalog). BOOTX64.EFI (amd64) / BOOTAA64.EFI (arm64). loader from compat /boot. |
| mkimg GPT | 2763-2768 | mkimg -s gpt -f raw -b rootfs/boot/pmbr -p freebsd-boot/bootfs:=…gptboot -p efi/efiboot0:=esp.img -p freebsd-ufs/ROOTFS:=rootfs.ufs -o NextBSD-${ARCH}-${IMG_DATE}.img (arm64: drop freebsd-boot, UEFI-only). |
| zip + sha256 | 2777-2780 | Native Linux. Unchanged. |
The ownership problem (non-root builder): the Linux runner cannot chown to root:wheel or set schg on disk. Solution — the FreeBSD make release pattern: feed makefs a -F METALOG mtree manifest so ownership/mode/flags are written into the UFS as metadata without ever chowning on disk. We preserve $WORK/METALOG across the whole assembly: every component install records into it via install -M METALOG; a reconciliation awk pass appends uid=0 gid=0 mode=<stat> defaults for overlay/base/db files lacking entries, with explicit overrides (.ko 0555, linker.hints 0444, spwd.db 0600, sshd privsep dirs). The db-regen outputs are appended to METALOG before makefs runs.
Tool sourcing: build makefs/mkimg/kldxref from /usr/src and host pwd_mkdb/cap_mkdb in the build job (first PR); strongly recommend a follow-up baking these into nextbsd-kernel-toolchain (compat wants them too) so nextbsd's PR stays workflow+script only.
build job — full replacement. Delete the entire vmactions/freebsd-vm "build ISO inside FreeBSD VM" block plus the distfiles cache that existed only for the VM path. New job:
runs-on: ubuntu-24.04; env: CCACHE_DIR: ${{github.workspace}}/.ccache; strategy.fail-fast:false; matrix amd64 + (gated) aarch64.container.image: ghcr.io/nextbsd-redux/nextbsd-kernel-toolchain:${{matrix.target}}-${{(github.event.client_payload.fbsd_sha || inputs.fbsd_sha) && format('fbsd-{0}', github.event.client_payload.fbsd_sha || inputs.fbsd_sha) || 'latest'}}.ccache-nextbsd-${{matrix.target}}-${{github.run_id}}, restore-keys ccache-nextbsd-${{matrix.target}}-) → ccache -o max_size=10G; ccache -z → download base/kernel-obj/modules (cross-repo, run-id + DISPATCH_TOKEN; modules continue-on-error) → build host tools → make.py toolchain && _includes → SKIP_VM=1 ARCH=$TARGET_ARCH sh build-cross.sh → cross cmake (libdispatch+ICU) → assemble image → ccache -s → upload NextBSD-${{matrix.target}}-${{steps.meta.outputs.date}}.test job — extend, don't restructure. Already ubuntu-latest + qemu + expect + OVMF (never vmactions). Add a 2-cell matrix: amd64 → qemu-system-x86 + ovmf; aarch64 → qemu-system-arm + qemu-efi-aarch64 (AAVMF, gated). tests/boot-test.sh unchanged; pass ARCH to select the qemu binary/firmware.
release job — collapse to single non-matrix job (the in-file NOTE warns of a matrix race on gh release delete): needs:[build,test], if: push main, download all NextBSD-* artifacts merge-multiple, delete+publish continuous once via softprops/action-gh-release@v2.
Dispatch rewiring. nextbsd already consumes base-updated. Add one line to compat's base-updated dispatch: -f "client_payload[compat_run_id]=${{github.run_id}}" (and ideally [kernel_run_id]/[modules_run_id]) so nextbsd downloads the exact matching artifacts by run-id rather than gh run list latest-success (avoids ABI/KBI skew). Thread fbsd_sha through to pin the toolchain image tag to the same source the kernel/base were built from. Add fbsd_sha/run-id inputs to workflow_dispatch for manual pins.
The parity oracle is tests/boot-test.sh, unchanged — the ~80-marker expect contract that gates today's vmactions image. Because it is byte-identical and discovers the .img member purely by extension (boot-test.sh:30), a green run on the Linux-assembled image is definitionally boot parity: same firmware (OVMF), same expect markers, same image-name shape; the only variable changed is *where* the image was assembled.
Static gates (before the ~17-min qemu boot — fail fast):
gdisk -l/sgdisk -p confirm the 3 GPT partitions (freebsd-boot, efi, freebsd-ufs/ROOTFS); file -s the ufs partition shows "Unix Fast File System"; mtools/loopback the ESP confirms /EFI/BOOT/BOOTX64.EFI.cc and cmake host-leak risks as *build* failures): llvm-readelf -h over rootfs/{bin/sh, sbin/launchd, usr/sbin/pwd_mkdb, usr/lib/system/libsystem_kernel.so, lib/libdispatch.so.0, lib/libicucore.so.74} asserting EI_OSABI=FreeBSD and e_machine matches target.llvm-readelf -d over the rootfs, resolving against /usr/lib + /usr/lib/system (replaces the deleted chroot ldd self-tests).icudt74_dat symbol present; ucol_open unrenamed.Dynamic gate (authoritative): boot the artifact under qemu+OVMF. Markers map precisely to assembly defects:
| Marker | Proves | A failure points to |
|---|---|---|
| loader OK prompt within 60s | UEFI ESP + loader.efi placement | mkimg / ESP |
FreeBSD banner + login: | freebsd-ufs root mounts rw, label=ROOTFS, linker.hints/mach.ko preload | makefs/UFS/metalog |
| MACH-SMOKE-OK (232) | mach.ko indexed + loaded | kldxref / linker.hints / KBI |
| LIBSYSTEM-KERNEL-OK | libmach links | libmach cross |
| LIBDISPATCH-OK (313) | dispatch roundtrip | libdispatch cross |
| LIBDISPATCH-MACH-OK (324) | HAVE_MACH protocolServer + -lsystem_kernel | MIG codegen |
| PAM-LOGIN-OK | offline pwd_mkdb/cap_mkdb db format | offline db regen |
halt -p → qemu exit 0, "boot-test PASSED" | full contract | — |
release publishes only when build+test both pass on push to main.
Land in dependency order so each step is independently green before the next. Use a per-component fallback flag so the two cmake nuts land last.
build job container + matrix; ccache; download base/kernel-obj/modules; extract base to $WORK/rootfs. Add SKIP_VM guard to build.sh (skip pkg front-half 65-340, all chroot blocks). *Gate:* artifacts present, sysroot has usr/include/stdint.h.mig -version, makefs --help run on the runner.mach.ko ELF for target via llvm-readelf -h; KERNBUILDDIR matched.libsystem_kernel.so in /usr/lib/system; .defs staged.cc→$(CC), ELF-ABI gate per suite).-jN). *Gate:* icudt74_dat symbol, unrenamed ucol_open.-F, offline db). *Gate:* static GPT/ESP/UFS checks.| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
5 GNU-make suites silently emit Linux ELF (hardcoded cc) → unbootable load-bearing tools (/bin/sh, pwd_mkdb, cap_mkdb) | Med | High | sed cc→$(CC)/install→$(INSTALL) in build-cross.sh; per-suite llvm-readelf -h ELF-ABI gate turns a missed recipe into a *build* failure, not a boot failure. |
| HAVE_MACH silently 0 (mach.h deferred) → libdispatch builds non-Mach stub, MIG never runs | Med | High | Install mach/mach.h before configure; hard post-configure grep HAVE_MACH 1 assert; curated mach.h if umbrella over-includes. |
ICU decoder cross-compiled → Exec format error → empty data blob | Med | High | Option (iii): host pre-decode + -DICU_PACKAGED_BIN_PATH, delete add_executable. Validate icudt74_dat ~40 MB OBJECT post-install. |
| MIG host/cross split wired backwards → empty *Server.c/*User.c, every daemon fails | Med | High | migcom native (no --target); MIGCC='clang --target=… --sysroot=…'; libmach .defs first; smoke-compile one generated *User.c for target before proceeding. |
find_library(system_kernel NO_DEFAULT_PATH) bypasses sysroot reroot | Med | Med | FIND_ROOT_PATH_MODE_LIBRARY=ONLY; if it trips, patch PATHS to ${CMAKE_SYSROOT}/usr/lib/system. Verify at first configure. |
Cross-repo ABI mismatch (gh run list latest pulls wrong fbsd_sha) → KBI/libc-libthr skew | Med | High | Thread fbsd_sha + explicit run-ids through base-updated; pin image tag + download run-ids to them; mach.ko KERNBUILDDIR from matching kernel-obj. |
makefs -F metalog gaps → builder-owned root:wheel/0555 files → sshd/login/PAM misbehave at boot | Med | Med | Drive ALL installs through install -M METALOG; reconciliation awk pass for un-tracked paths; boot markers (PAM-LOGIN-OK) are the ownership oracle. |
| makefs/mkimg cross-for-host endian/CRC bug → OVMF/loader rejects image (no serial marker) | Low | High | These are FreeBSD's OWN build-host tools (run during every buildworld/make release) — same config FreeBSD ships. Static gdisk -l/file -s gate before qemu. |
| ICU C++17 picks up host libc++ → Linux-ABI object | Low | High | FIND_ROOT_PATH_MODE_INCLUDE=ONLY + --target=freebsd confines to sysroot libc++. Validate DT_NEEDED = FreeBSD libc++.so.1. |
| OpenSSH cross-configure runs target binaries in tests | Med | Med | --host=<arch>-unknown-freebsd15.0 + cached config.cache answers for run-tests. |
| makefs/mkimg/kldxref not in image → per-run build time + failure surface | Low | Low | Build once from /usr/src; follow-up: bake into toolchain image (compat wants them too). |
| arm64 has no compat base sysroot | High (today) | Low | Land amd64-first; gate aarch64 build leg until nextbsd-base-arm64 ships. Toolchain files already arch-parameterized → matrix add. |
OOM regression if icu_packaged_data.cpp re-added | Low | High | Already removed from all source lists; .incbin bounds peak ~150 MB; keep the -j2 cap *dropped* and monitor RSS. |
nextbsd repo (/tmp/nb):
| Path | Change |
|---|---|
.github/workflows/build.yml | Rewrite build job: drop vmactions/freebsd-vm + distfiles cache; add container: toolchain image + amd64/aarch64 matrix; ccache restore (kernel-repo key scheme); 3 cross-repo download-artifact steps (base/kernel-obj/modules, repository+run-id+DISPATCH_TOKEN); host-tool build step; make.py toolchain/_includes; build-cross.sh invocation; cross-cmake step; Linux image-assembly step; upload NextBSD-<arch>-<date>.img.zip. Extend test to per-arch qemu/firmware matrix. Collapse release to single non-matrix gather-and-publish. Add fbsd_sha/run-id workflow_dispatch inputs. |
build-cross.sh (NEW) | Top-level driver: dependency-ordered component list; per-comp make.py … buildenv BUILDENV_SHELL=<script> with DESTDIR=$WORK/rootfs + install -N -U -M METALOG; MIGCOM(host)/MIGCC(cross) split; sed cc→$(CC) for the 5 GNU-make suites; cross-cmake invocation for the 2 cmake comps. Sources existing build.sh recipes minus every chroot/ldconfig/ldd/mig self-test. |
build.sh | Add SKIP_VM guard: skip pkg front-half (65-340) + all chroot blocks + in-chroot cmake; image-assembly tail (2675-2780) → host binaries, no chroot: chroot kldxref→offline kldxref (645,653); chroot pwd_mkdb/cap_mkdb→offline -d/-p (2675-2676); readelf→llvm-readelf (2705); makefs ffs + -F $WORK/METALOG (2721); introduce $WORK/METALOG + reconciliation pass; parametrize ESP name (BOOTX64/BOOTAA64) + mkimg partition set on $ARCH; chflags noschg→rm. Drop host-coreutils/etc copy hacks (129-136,175-191). |
make-mach-kmod.sh | Host-cross mode: wrap make -C src/mach_kmod SYSDIR=/usr/src/sys KMODDIR=/boot/kernel KERNBUILDDIR=<kernel-obj>/sys/NEXTBSD in make.py buildenv --cross-bindir=$CCACHE_CROSS_BINDIR; drop chroot kldxref. |
cmake/freebsd-cross-toolchain.cmake (NEW) | Shared CMake toolchain file (§5): CMAKE_SYSTEM_NAME=FreeBSD, arch-param SYSTEM_PROCESSOR/triple, ccache-wrapped clang-19, CMAKE_SYSROOT+FIND_ROOT_PATH=$WORK/rootfs, FIND_ROOT_PATH_MODE PROGRAM=NEVER / LIBRARY=ONLY / INCLUDE=ONLY, CMAKE_PROGRAM_PATH+=hosttools/bin, -fuse-ld=lld. |
src/libdispatch/src/CMakeLists.txt | Likely no change; patch only if find_library(system_kernel … NO_DEFAULT_PATH) (~199) defeats sysroot reroot → PATHS ${CMAKE_SYSROOT}/usr/lib/system. |
src/swift-foundation-icu/icuSources/common/CMakeLists.txt | Fix freebsd_decode_icu_chunks split (option iii): delete add_executable (268) + add_custom_command (272-286) + add_custom_target/add_dependencies (295-297); accept -DICU_PACKAGED_BIN_PATH; keep uvernum.h extraction (304-312) + configure_file .S (315-318) + target_sources (334-335). |
src/swift-foundation-icu/PORTING_README.md | Update Build section: native cross cmake + pre-decoded blob; note -j2/8GB comments obsolete. |
Sibling repos (small edits):
| Path | Change |
|---|---|
nextbsd-freebsd-compat/.github/workflows/build.yml | One line: add -f "client_payload[compat_run_id]=${{github.run_id}}" (and ideally kernel/modules run-ids) to the base-updated dispatch. |
nextbsd-kernel-toolchain/Dockerfile.amd64 + .arm64 | Add cmake ninja-build byacc flex to apt install; follow-up: bake host-built makefs/mkimg/kldxref onto PATH. |
The PR is done when, on push to main:
vmactions/freebsd-vm step is deleted; build.sh's pkg-bootstrap front-half and every chroot/ldconfig/ldd/mig self-test are removed under SKIP_VM; no pkg install anywhere; cmake/ninja come from apt, clang from the image.build job runs to green inside ghcr.io/nextbsd-redux/nextbsd-kernel-toolchain:amd64-fbsd-<sha>, cross-building all 32 components + mach.ko + the two CMake libs, with ccache -s showing nonzero cached compiles.EI_OSABI=FreeBSD + correct e_machine (llvm-readelf -h); DT_NEEDED closure resolves; libdispatch config_ac.h HAVE_MACH 1; ICU icudt74_dat present + ucol_open unrenamed; gdisk -l/file -s confirm GPT(freebsd-boot+efi+freebsd-ufs/ROOTFS)+UFS+ESP/BOOTX64.EFI.NextBSD-amd64-<date>.img.zip is byte-shape-identical (name/extension/zip+sha256) to the former vmactions output.tests/boot-test.sh boots that image under qemu-system-x86_64 + OVMF and emits the full marker chain — loader prompt → login: → MACH-SMOKE-OK → LIBSYSTEM-KERNEL-OK → LIBDISPATCH-OK → LIBDISPATCH-MACH-OK → … → PAM-LOGIN-OK → halt -p → qemu exit 0 → "boot-test PASSED".release publishes the continuous release exactly once (single non-matrix job).releng/15.0 → toolchain → {kernel → modules, compat} → nextbsd completes end-to-end with fbsd_sha + run-ids pinned so the ingested base/kernel/modules match the toolchain image tag.nextbsd-freebsd-compat ships nextbsd-base-arm64 (toolchain files + matrix already arch-parameterized; test job's AAVMF path stubbed).Net result: everything builds on a native Linux runner via clang cross-compilation, zero FreeBSD packages, nothing in a chroot, the bootable UFS/GPT image assembled on Linux, and qemu used only to boot-test — with the proven sibling pattern carrying ~28 of 30 bmake components and mach.ko at low risk, and the two CMake nuts de-risked to mechanical toolchain-file + host-tool-split work.
Generated from a 9-agent analysis workflow (4 investigators reverse-engineering the proven cross-build pattern across toolchain/kernel/modules/compat + a full component map, 4 designers solving libdispatch, ICU, the CI rewrite, and Linux image assembly, 1 synthesizer). Feasibility is treated as proven by the sibling repos; the plan focuses on execution order, the two net-new CMake toolchain files (libdispatch, ICU), and boot-contract parity. Cross-repo testing improvements (kernel-PR validation, coupled changes) are deliberately out of scope — a separate follow-on effort.