How to boot-test coupled changes that span two repos before either merges — the motivating case being a nextbsd-kernel PR that raises FreeBSD’s dynamic syscall-slot limit plus a nextbsd PR that drops mach.ko’s syscall multiplexor in favour of individually-numbered Mach trap slots. Five CI strategies compared with realistic time-to-test for each, in today’s vmactions VM world and the near-future native-cross world.
Status — June 2026 What shipped (and how it diverged from the recommendation below).
continuous release per producer (kernel / modules / compat) rather than the versions.lock immutable-pin variant. nextbsd's three ingests now gh release download continuous instead of the gh run list --branch main --status success --limit 1 scrape, which closes the broken-/half-cascaded-main ingest hole (GAP#6): only a green main build ever updates continuous. (No per-SHA versions.lock pin yet — the rolling tag is the pointer.)push:false) — closing GAP#1's “kernel has no pull_request trigger”. Invariant: a PR never updates continuous anywhere (publish/release require refs/heads/main).nextbsd-kernel”: instead of rebuilding a full ISO via override inputs, it injects the PR's freshly-built kernel into the latest continuous NextBSD ISO (FreeBSD-VM mdconfig mount of the UFS root) and boots it under qemu with nextbsd's own boot-test.sh. PR-only; does not gate publish.releng/15.0, so kernel KBI is stable and the shipped mach.ko + modules keep loading against a freshly-built kernel. The live question became “does my kernel patch still boot?” — answered by the single-repo smoke test above, not cross-repo coupled pre-merge testing.*_run_id/*_ref inputs, the versions.lock immutable pin + pin-bump bot, the /integrate bot, the monorepo (C1/C2), and the D meta-workflow. Revisit if/when a genuinely coupled cross-repo change lands. The options analysis below is preserved as the decision record.Two changes must be boot-tested together before either merges. PR #1 in nextbsd-redux/nextbsd-kernel adds a patch (to patches/series) that widens FreeBSD's fixed band of dynamically-claimable syscall slots (today only the ~10 lkmnosys slots 210-219 exist, and NextBSD's mach.ko already burns all of them). PR #2 in nextbsd-redux/nextbsd drops the multiplexor: it deletes the mach_trap_mux_sysent / sys_mach_trap_mux_trap switch in src/mach_kmod/src/mach_traps.c (which today folds task_set_special_port=op2, host_set_special_port=op1, mach_port_move_member=op3, event-bell ops behind one slot), adds per-trap wire_one() calls in src/mach_syscall_wire.c, and rewrites src/libmach/mach_traps.c to resolve_syscall(\"task_set_special_port\") directly instead of resolve_syscall(\"mach_trap_mux\")+op. The coupling is mechanical, not conceptual: if the userland/module side ships individual syscalls but the kernel slot limit is still 10, the extra wire_one() calls fail (kern_syscall_register returns ENFILE), the sysctls publish -1, resolve_syscall returns -1, and syscall(-1) faults — every task_set_special_port / host bootstrap / port_move_member call breaks. The kernel patch alone is inert (stock kernel boots fine via the mux). The failure mode is invisible to compilation and to single-repo CI: only a live kernel booting the combined ISO proves the raised cap actually granted runtime slots. So the strategy must build {nextbsd-kernel@syscall-limit} x {nextbsd@mux-drop} into ONE ISO and boot it under QEMU, asserting TASK-SPECIAL-PORT-OK + HOST-BOOTSTRAP-OK + BOOTSTRAP-REMOTE-OK (with smoke-sysctls.sh as a cheaper pre-gate: each of mach.syscall.task_set_special_port/.host_set_special_port/.mach_port_move_member returns a dedicated >=0 slot and mach.syscall.mach_trap_mux is gone). The complication: the IMG only builds today inside a vmactions FreeBSD VM (~17 min, the long pole and the only repo that boots), while kernel/modules/compat already cross-build fast on Linux. A crossbuild migration making the IMG build natively on Linux (~5 min + ccache) is in-flight, which flips the economics — so every option is evaluated in BOTH today's VM world and the near-future native-cross world.
| Option | Family | Verdict | Best when |
|---|---|---|---|
| A. Continuous releases on every repo + PRs consume latest RELEASED artifact (versions.lock latest-green pin) | multi-repo | Partial — keep the pin | Adopt as the STABILITY BACKBONE, not the pre-merge coupling answer: when you want to harden the cascade against ingesting broken/half-cascaded main artifacts and gain immutable, reproducible, quarantined consumption. Best paired with B (or later D) which supply the actual pre-merge coupling. The versions.lock latest-green pin is the piece worth adopting NOW even if you skip the rest of A. |
| B. Artifact-override workflow_dispatch inputs (nextbsd gains kernel_run_id/base_run_id/modules_run_id; downstream PR points at an upstream PR/branch build) | multi-repo | Adopt now | Adopt NOW, in today's VM world, as the lowest-effort path to pre-merge coupled validation — idiomatic to these repos and ~80% present. Ideal when coupled changes are occasional and a 2-ref (kernel+nextbsd, optionally +modules/base) override covers the case; graduate to D once combinations grow beyond a couple of refs, and layer A's latest-green pin + a merge queue as the safety rail. |
| C1. Full monorepo (toolchain + kernel + modules + compat + nextbsd in one repo, path-filtered workflows) | monorepo | Alt (after crossbuild) | Adopt as the LONG-TERM TARGET strictly AFTER the crossbuild migration lands and the IMG builds natively on Linux. It is the cleanest answer to coupled changes precisely because the only reason the repos were split — the VM/artifact handoff boundary — is dissolved by native cross-build. Sequence: ship crossbuild -> use B as the bridge meanwhile -> migrate to this monorepo + path filters + merge queue. Prefer C1 over C2 only once the external toolchain image boundary proves more annoying than valuable. |
| C2. Partial monorepo (kernel + modules + compat + nextbsd merged; toolchain stays a separate GHCR image, pinned by digest) | hybrid | Target (after crossbuild) | Prefer over C1 when the toolchain genuinely rebuilds on a different (slower, source-bump-driven) cadence than the OS source — which is the case here (toolchain only rebuilds on releng/15.0 bumps) — and when you want the cross-compiler pinned by immutable digest as the reproducibility anchor. Also the lower-risk FIRST monorepo step: migrate to C2, and only fold the toolchain in later (-> C1) if the external image boundary proves more annoying than valuable. |
| D. Hybrid integration meta-workflow over {repo:ref} pairs (+ latest-green pin + merge queue) | hybrid | Keep thin / on-demand | You want pre-merge boot-test of arbitrary coupled cross-repo branch combinations WITHOUT a monorepo migration and WITHOUT giving up per-repo release isolation — the realistic near-term answer that handles the syscall+mux pair today and scales to N-ref coupling, then layers a latest-green pin (adopt now, cheap, fixes the broken-main-ingest hole) and a merge queue (adopt on the ISO repo once boot tests are fast post-crossbuild) as safety rails on top. Adopt D when 2-ref B overrides stop sufficing, or directly if you anticipate N-ref coupling and want the recorded-ref-set ergonomics from day one. |
Wall-clock per build stage in three worlds: today’s vmactions VM, native cross-build cold (no ccache), and native cross-build warm (ccache hot). These are the building blocks behind every option’s time-to-test.
| Stage | Today (VM) | Cross cold | Cross warm |
|---|---|---|---|
| Toolchain container (GHCR pull, cached; full rebuild only on releng source bump) | ~1m pull in-VM; full rebuild ~6m (rare) | ~1m docker pull + type=gha layer cache; full rebuild ~6m (rare) | ~0.3m warm layer cache, no pull |
| Kernel build (nextbsd-kernel, Linux x86_64 cross via toolchain image — same job in every world) | 7m45s cold / 3m09s warm (already a Linux cross job, NOT in the VM) | 7m45s (465s) cold | 3m09s (189s) warm, ccache hot |
| mach.ko relink (module half of the mux-drop, built against extracted /usr/src/sys) | ~1.5m (on the FreeBSD host at ISO-build time, inside the VM) | ~1m (Linux cross compile of one kmod, no ccache) | ~0.3m (ccache hot; tiny incremental relink) |
| Userland build — 32 comps incl. libmach (userland half of the mux-drop) | ~7m (dominant part of VM source-compile ~8.5m; libicu -j2 OOM-capped) | ~5m (native Linux cross, no ccache) | ~1.5m (ccache hot) |
| Image assembly (pkg install + makefs/mkimg + zip) | ~6.2m (pkg ~2.3m + zip ~3.9m, in-VM) | ~4m (native makefs/mkimg + zip) | ~3m (zip compression dominates; ccache-immune) |
| Boot test (qemu + expect, ~80 markers incl. TASK-SPECIAL-PORT-OK / HOST-BOOTSTRAP-OK / BOOTSTRAP-REMOTE-OK) | ~3m (qemu q35 nested inside the FreeBSD VM) | ~2.5m (qemu on Linux host, KVM) | ~2.5m (boot time fixed; ccache irrelevant) |
| VM-only overhead (vmactions spin-up + guest boot + copyback) — DISAPPEARS in native world | ~4m (VM boot ~2m + copyback ~2m) | 0m (no VM) | 0m (no VM) |
| Case | Today (VM) | Cross cold | Cross warm |
|---|---|---|---|
| Kernel-only PR — A (continuous releases + pin) | NOT pre-merge testable end-to-end; release-then-pin-bump path ~20m+ AND post-merge only | ~13.5m but still post-merge only (release+pin dance) | ~7m but still post-merge only |
| Kernel-only PR — B (artifact-override inputs) | ~28m: kernel cold 7m45s + dispatch nextbsd ISO reusing cached base/modules ~17m boot; one extra full ISO+boot | ~13.5m (kernel 7m45s + assembly 4m + boot 2.5m; toolchain cached ~1m) | ~6.5m (kernel 3m09s + assembly 3m + boot 2.5m) |
| Kernel-only PR — C1/C2 (monorepo + path filters) | N/A — blocked until crossbuild lands (no FreeBSD VM on the monorepo Linux runner) | ~13m (C1) / ~13m (C2): one PR, one run — kernel 7m45s + mach.ko 1m + (userland path-skipped) + assembly 4m + boot 2.5m | ~6m (C1) / ~5.5m (C2): kernel 3m09s + mach.ko 0.3m + assembly 3m + boot 2.5m; C2 slightly faster (toolchain never content-hash-evaluated, fixed pinned digest pull) |
| Kernel-only PR — D (meta-workflow {repo:ref}) | ~28m on-demand (same single-ISO cost as B; run by label/comment to bound VM cost) | ~13.5m (same as B; meta-workflow builds kernel ref + assembles + boots) | ~6.5m (same as B, generalized to N refs) |
| Coupled PR (syscall-limit kernel + mux-drop userland/mach.ko) — A | NOT possible pre-merge: requires RC tags of BOTH branches + pin bump before any combined boot — post-merge validation only | Still post-merge only; ~13.5m once both released and pinned | Still post-merge only; ~7m once released and pinned |
| Coupled PR — B (artifact-override inputs) | ~28m: kernel PR branch 7m45s (prior) -> dispatch nextbsd ISO with kernel_run_id + mux-drop in nextbsd PR -> VM build ~17m + boot ~3m. One extra dispatch. Needs GAP#4 fix (ref-aware module patch clone) | ~14.5m: kernel 7m45s ∥ userland 5m -> assembly 4m -> boot 2.5m | ~7m: kernel 3m09s ∥ userland 1.5m -> assembly 3m -> boot 2.5m |
| Coupled PR — C1/C2 (monorepo, one PR + one run) | N/A — blocked until crossbuild lands | ~14.5m: single PR rebuilds kernel 7m45s + mach.ko + userland (path-filtered) -> IMG -> boot. No cross-repo refs; coupling eliminated by construction | ~7m: kernel 3m09s ∥ userland 1.5m -> mach.ko 0.3m -> assembly 3m -> boot 2.5m. Lowest steady-state, best ergonomics |
| Coupled PR — D (meta-workflow over both refs) | ~28m on-demand (label/comment-triggered): builds kernel_ref + nextbsd_ref together, one recorded run, full VM ISO+boot | ~14.5m (same critical path as B/C; generalizes to a 3rd/4th coupled ref) | ~7m (cheap enough to run on every cross-repo-labeled PR) |
Time-to-test strings on each option are made consistent with this table. MODEL BASIS (real measurements): nextbsd VM build ~17m = userland ~7m + mach.ko ~1.5m + pkg ~2.3m + zip ~3.9m + VM boot ~2m + copyback ~2m; kernel 7m45s cold / 3m09s warm (already a Linux cross job); compat ~4m; boot test ~3m. Native cold/warm derived by removing VM/emulation overhead and applying ccache to compile stages only (zip and boot are fixed-cost, ccache-immune). KEY ASSUMPTIONS: (1) kernel/modules/compat ALREADY cross-build on the x86_64 Linux runner today; only the nextbsd ISO leg is VM-bound, so the entire 'today VM' penalty (~17m) lives in image-assembly+boot, and the kernel row is identical across all worlds. (2) Coupled critical path = kernel (7.75m cold / 3.15m warm) IN PARALLEL with userland (5m / 1.5m); the longer feeds assembly (4m/3m) -> boot (2.5m); hence coupled ~14.5m cold / ~7m warm (stages are NOT summed where they parallelize). (3) Boot is a fixed ~2.5-3m floor in every world (qemu+expect over ~80 markers, wall-clock bound by guest boot) — the irreducible floor of any boot-tested option; the cheapest sub-boot signal for THIS change is smoke-sysctls.sh. OPTION-LEVEL CONCLUSIONS: A never delivers pre-merge coupled testing at any speed (validates only after both halves are released) — keep it as the stability backbone (latest-green pin), not the coupling answer. B is the cheapest path to the goal and ~80% built (modules already has run_id+fbsd_sha; toolchain already publishes :<branch> tags and gates cascade on main); coupled cost = one extra full ISO+boot (~28m today, ~14.5m native cold, ~7m native warm); HARD CAVEAT GAP#4 — modules 'Apply patches to source' clones kernel MAIN with no ref, so for a coupled kernel-SOURCE change the patch source must be made ref-aware or mach.ko is KBI-desynced. C1/C2 give the ideal coupled ergonomics (one PR, one run, ~7m warm, lowest steady-state) but are BLOCKED in today's VM world. D matches B's per-run cost, generalizes to N coupled refs, and records the exact ref-set on each PR. Merge-queue + latest-green pin are a safety rail, not a coupling mechanism: adopt the pin NOW (replaces fragile 'gh run list --branch main --status success --limit 1' that can ingest a half-broken/half-cascaded main, fixing GAP#6 sha-coherence at the three nextbsd ingests); add the merge queue on the ISO repo once boot tests are fast (post-crossbuild), since it adds one re-validation boot per merge (~3m, noticeable today). ARM64 CAVEAT (GAP#5): every number is amd64 — compat is amd64-only and nextbsd ingests amd64-only, so arm64 has ZERO end-to-end/boot coverage today regardless of option. FORK-PR CAVEAT (GAP#7): all cross-repo reads use secrets.DISPATCH_TOKEN, unavailable to fork pull_request runs, so B/D auto-trigger is limited to same-repo branches until a GitHub App token / pull_request_target path is added.
Every repo in the cascade cuts an immutable, versioned release on each main merge instead of leaving artifacts as ephemeral 90-day run outputs. toolchain already publishes immutable :<arch>-fbsd-<sha> GHCR tags and a mutable :<arch>-latest; under A you add a date/semver release tag so the image is addressable by a stable name. kernel, modules and compat gain a 'release' job (modules already has a dormant one gated client_payload.release=='true' — you flip it on for every main build) publishing per-arch GH releases (kernel-<date>-<sha>-<arch>, etc.). nextbsd keeps its rolling 'continuous' release. The keystone is a checked-in versions.lock in nextbsd recording the last fully-integrated boot-green set {toolchain_tag, kernel_release, modules_release, base_release}. The nextbsd ISO build's three ingest steps STOP doing 'gh run list --branch main --status success --limit 1' and instead 'gh release download <tag-from-versions.lock>'. A pin-bump bot opens a PR editing versions.lock whenever a new upstream release passes a green integration boot; merging that PR promotes the artifact set. Downstreams are always pinned to an immutable, known-good release rather than racing latest-main-success scraping (which can ingest a half-cascaded/freshly-broken main artifact — GAP#2/GAP#6).
Coupled change (syscall + multiplexor) flow: The syscall-limit (kernel) + mux-drop (nextbsd src/mach_kmod + src/libmach) pair is INHERENTLY post-merge under pure A: pinning consumes only RELEASED tags, and you cannot release an unmerged branch without polluting the release namespace. To co-test pre-merge you must cut throwaway PRE-RELEASE tags of each branch and a temporary pin: (1) branch+push the kernel patch so a path-filtered push on patches/** builds it; (2) dispatch a release-flagged build to promote that branch build to a prerelease the pin can name (requires adding a release/prerelease input to kernel); (3) because the modules patch clone is hardcoded to kernel MAIN (GAP#4), cut a matching modules prerelease pointing its patch clone at the kernel branch — modules has no such ref input, so pure A cannot produce a coherent mach.ko for an unreleased kernel without that fix; (4) open the nextbsd mux-drop PR and temporarily edit versions.lock to point kernel_release/modules_release at the prerelease tags; the PR boot test gh-release-downloads the prerelease set, boots the combined ISO and asserts TASK-SPECIAL-PORT-OK + HOST-BOOTSTRAP-OK + BOOTSTRAP-REMOTE-OK; (5) after green, UN-pin (revert versions.lock) before merge, re-cut REAL releases on each merge, then bump versions.lock for real. Net: >=3 repos and 2-3 PRs plus prerelease bookkeeping — clumsy, multi-PR, and still hits GAP#4.
Today (VM): NOT pre-merge end-to-end — a kernel-only PR is observable only by promoting a prerelease and re-pinning the ISO build, so realistically ~20m+ wall clock dominated by the vmactions ISO build+QEMU boot, PLUS manual prerelease+pin ceremony, and still post-merge in spirit. Native-cross: ~7m compute once a prerelease+pin exists, but the human prerelease/pin ceremony still dominates and keeps kernel-PR iteration slow.
Today (VM): NOT possible pre-merge without throwaway RC tags of BOTH branches + a temporary pin edit/revert; the combined boot itself is one ~28m-class VM ISO+boot, but the prerelease+pin dance makes real iteration high-friction and post-merge in nature. Native-cross: ~7m once both are released and pinned, but the multi-prerelease + pin-edit/revert overhead keeps real coupled iteration high-friction and still post-merge.
Ratings: coupling ease low speed low infra simplicity medium blast-radius control high release flexibility high
Generalize the override seam that ALREADY exists on nextbsd-kernel-modules (workflow_dispatch run_id + fbsd_sha, resolving client_payload.run_id || inputs.run_id and downloading cross-repo artifacts with DISPATCH_TOKEN) to the whole cascade. Three edits: (1) add a pull_request trigger AND a short boot/sysctl smoke to nextbsd-kernel so a kernel branch/PR builds the kernel + (on source change) mach.ko and produces nextbsd-kernel-<arch> / kernel-obj-<arch> under a known run_id — toolchain already publishes branch-scoped :<arch>-<branch> tags and gates all downstream dispatch on ref_name=='main', so branch builds are already addressable AND non-cascading; (2) add kernel_run_id / base_run_id / modules_run_id (+ matching *_ref) inputs to the nextbsd ISO build: when set, each replaces the 'gh run list --branch main --status success --limit 1' lookup with 'gh run download --repo <repo> --run-id <id>'; when unset, today's latest-main behavior is preserved; (3) FIX GAP#4: make the modules 'Apply patches to source' step clone the kernel at a passed kernel_ref instead of hardcoded main, so an override run patches module SOURCE against the same kernel branch the obj came from. Optionally add a /integrate kernel#NN PR-comment bot that resolves the head run_id, dispatches the ISO build, and posts boot results to both PRs. Override runs never publish 'continuous' and never cascade, so a pinned PR build can't touch prod artifacts.
Coupled change (syscall + multiplexor) flow: Boot-test the syscall-limit (kernel) + mux-drop (nextbsd) pair BEFORE either merges: (1) create the kernel branch with the lkmnosys-band patch and push (path-filtered push on patches/** auto-builds; or PR once pull_request is added); (2) grab that build's KERNEL_RUN_ID; (3) with the GAP#4 fix in effect, dispatch modules against the SAME kernel branch (gh workflow run build.yml --repo nextbsd-kernel-modules -f run_id=$KERNEL_RUN_ID -f kernel_ref=syscall-limit) so mach.ko's source matches the obj, capture MODULES_RUN_ID; (4) open the nextbsd mux-drop PR (wire_one() per-trap + deletion of the mach_trap_mux switch in mach_traps.c; libmach resolve_syscall per-name); (5) dispatch the nextbsd ISO build pointing at the unmerged runs (gh workflow run build.yml --repo nextbsd --ref mux-drop -f kernel_run_id=$KERNEL_RUN_ID -f modules_run_id=$MODULES_RUN_ID; base_run_id omitted -> latest-main); the build downloads the pinned kernel/modules, builds libmach+mach.ko from the branch, bakes the ISO, boots it; (6) watch the boot assert TASK-SPECIAL-PORT-OK + HOST-BOOTSTRAP-OK + BOOTSTRAP-OK + BOOTSTRAP-REMOTE-OK + MACH-SMOKE-OK/LIBSYSTEM-KERNEL-OK; a cheaper pre-gate is smoke-sysctls.sh asserting each of mach.syscall.task_set_special_port/.host_set_special_port/.mach_port_move_member returns a dedicated >=0 slot and mach.syscall.mach_trap_mux is GONE. One extra dispatch (or one /integrate comment) yields the full boot-tested coupled ISO.
Today (VM): kernel compile+smoke alone ~3m09s warm / ~7m45s cold on the Linux runner (the new local kernel signal B introduces); a full end-to-end boot of a kernel-only change via the ISO override costs the nextbsd VM leg, ~28m. Native-cross: kernel compile+smoke ~3m09s warm; full boot-tested kernel PR ~6.5m warm / ~13.5m cold.
Today (VM): ~28m wall clock, dominated by the single vmactions ISO build+QEMU boot (~17m + ~3m boot); the kernel and modules override legs run fresh but cheap (kernel 7m45s, modules ~3m) and precede the ISO dispatch, which downloads their pinned artifacts rather than rebuilding the kernel — exactly ONE slow VM boot per iteration. Native-cross: ~14.5m cold / ~7m warm end-to-end (kernel ∥ userland -> assembly -> boot) — cheap enough to run on every cross-repo-labeled PR and the natural stepping stone to D.
Ratings: coupling ease high speed medium infra simplicity high blast-radius control high release flexibility medium
After the IMG cross-builds on Linux, collapse all five repos into one tree and let path-filtered workflows decide what rebuilds. Layout: /toolchain/ (Dockerfile.amd64/arm64), /kernel/patches/ (series + the syscall-limit patch) + /kernel/config/NEXTBSD, /src/ (the ~33 userland comps incl. mach_kmod, libmach, libdispatch, launchd, libxpc...), /compat/, /image/ (build.sh, overlays, pkglist), /tests/ (boot-test.sh + the freebsd-launchd-mach markers), /.github/workflows/, /versions/freebsd.lock (FREEBSD_BRANCH=releng/15.0 + pinned SHA — freebsd-src stays EXTERNAL, cloned at build time exactly as toolchain's Dockerfile does today, NOT vendored, so the repo stays ~325M not multi-GB). ONE build.yml with dorny/paths-filter computes the affected set, all jobs chained via needs:: toolchain-image (on toolchain/ OR freebsd.lock else resolves to current :latest digest); kernel (on kernel/ -> build inside toolchain image, emits kernel-obj as in-run artifact); modules (on kernel/ OR src/mach_kmod/ -> build mach.ko against THIS run's kernel-obj, killing GAP#4 by construction); userland (on src/ -> rebuild only affected comps, ccache-scoped); compat (on compat/ OR freebsd.lock); image (ALWAYS, the integration point: assemble from whichever of {kernel-obj, modules, userland, compat} this run produced, falling back to cached latest-green for untouched legs) -> boot-test asserts TASK-SPECIAL-PORT-OK + HOST-BOOTSTRAP-OK + BOOTSTRAP-REMOTE-OK. All cross-repo gh download / DISPATCH_TOKEN / repository_dispatch machinery is DELETED — artifacts flow via in-run needs:/download-artifact, so GAP#6 and GAP#7 both vanish.
Coupled change (syscall + multiplexor) flow: The syscall-limit + mux-drop pair becomes ONE atomic PR touching three subtrees in one commit graph: /kernel/patches/ (lkmnosys-band increase + series entry), /src/mach_kmod/ (drop mach_trap_mux_sysent in mach_traps.c, add per-trap wire_one() in mach_syscall_wire.c), and /src/libmach/mach_traps.c (resolve_syscall('task_set_special_port') instead of the mux op-dispatch). On push, paths-filter flags kernel/ AND src/mach_kmod/ AND src/libmach/** -> kernel rebuilds with the new slot band, modules rebuilds mach.ko against THAT kernel-obj (same run, same SHA), userland rebuilds libmach, image assembles them into one ISO, boot-test asserts the markers. NO ref-pinning, NO run_id wrangling, NO patch-clone-against-main desync (GAP#4 cannot occur — modules build against the PR's own kernel tree). One PR, one CI run, one boot test proves the slots are claimed at runtime — the textbook case the monorepo collapses into a single unit of change.
~6m warm / ~13m cold (POST-crossbuild only). kernel/ triggers: kernel rebuild (~3m09s warm / 7m45s cold; toolchain image a cache hit), modules rebuild (~0.3m warm, kernel/ affects mach.ko), image assemble from cached-green userland/compat (~3m warm / 4m cold), QEMU boot (~2.5m). Userland/compat SKIPPED by path filter. BLOCKED in today's VM world (the IMG still needs the vmactions FreeBSD VM, which the monorepo Linux runner does not have).
~7m warm / ~14.5m cold (POST-crossbuild only). The coupled PR triggers kernel + modules + libmach(userland) rebuilds (kernel ∥ userland), then the single image assemble + boot — still ONE run, ONE boot, because all legs converge in-run. C's win is not raw minutes (similar to B/D native) but that it is a single self-contained run with ZERO cross-repo coordination. N/A in today's VM world.
Ratings: coupling ease high speed high infra simplicity high blast-radius control low release flexibility low
Same as C1 but the toolchain stays its OWN repo, still publishing the GHCR cross-toolchain image (:<arch>-latest, :<arch>-fbsd-<sha>) exactly as today. The monorepo consumes that image as a job container, pinned by /versions/toolchain.lock (image digest) + /versions/freebsd.lock (releng SHA). Layout drops the /toolchain/ subtree; kernel/, src/, compat/, image/, tests/ are identical to C1. The path-filtered job graph is the same EXCEPT the toolchain-image job is replaced by a 'resolve-toolchain' step that reads versions/toolchain.lock and pulls the pinned digest — a toolchain bump is a one-line lockfile PR (or a repository_dispatch from the toolchain repo that opens a bump PR), not an in-tree image rebuild. kernel/modules/userland/compat/image jobs are byte-identical to C1. Rationale: the toolchain rebuilds rarely (only on releng/15.0 source bumps), is the slowest single artifact (per-arch docker build), and has the cleanest API boundary (a versioned container digest), so keeping it external avoids dragging heavy infrequent image builds into every coupled PR's tree and preserves the one boundary that genuinely benefits from independent cadence + GHCR's immutable digest addressing.
Coupled change (syscall + multiplexor) flow: IDENTICAL to C1 for the syscall+mux pair: it spans kernel/patches/, src/mach_kmod/, src/libmach/ — all inside the monorepo — so it is still ONE atomic PR + ONE CI run + ONE boot test, with the same GAP#4/#6/#7 elimination. The toolchain being external is irrelevant to this pair because the pair never touches the toolchain. The ONLY coupled change this variant does NOT collapse into a single PR is a kernel change requiring a SIMULTANEOUS toolchain bump (e.g. a new clang feature): that becomes a two-step — land/dispatch the toolchain image, then bump versions/toolchain.lock in the monorepo PR. Rare; the syscall scenario is unaffected.
~5.5m warm / ~13m cold (POST-crossbuild only) — slightly FASTER than C1 because the toolchain image is never content-hash-evaluated in-run; it is a fixed pinned digest pull (cache hit). kernel ~3m09s warm + modules ~0.3m + image assemble ~3m + boot ~2.5m, userland/compat path-skipped. BLOCKED in today's VM world (same hard prerequisite as C1).
~7m warm / ~14.5m cold (POST-crossbuild only), essentially the same as C1 minus any toolchain-image evaluation overhead. The coupled syscall+mux PR rebuilds kernel + mach.ko + libmach against the pinned toolchain digest, assembles one ISO, boots once. N/A in today's VM world.
Ratings: coupling ease high speed high infra simplicity medium blast-radius control low release flexibility medium
Keep all five repos separate exactly as today, but add ONE new dispatchable workflow, integration.yml, in nextbsd (or a tiny nextbsd-ci meta-repo for clean token scope and no path-filter collisions). It is a workflow_dispatch (+ repository_dispatch type: integrate, + PR-comment bot) workflow whose inputs are a SET of {repo:ref} pins: toolchain_ref, kernel_ref, modules_ref, compat_ref, nextbsd_ref (each a branch name, PR head ref, or sha; blank = current main/latest-green). It runs the WHOLE chain on demand for that exact ref-set, fresh and non-cascading: job toolchain (builds from toolchain_ref or reuses the ghcr :<arch>-<branch> tag, exporting an image the later jobs use as their container); job kernel (needs: toolchain — checks out kernel@kernel_ref, cross-builds, uploads kernel-<arch> + kernel-obj-<arch> AS ARTIFACTS OF THIS RUN, so everything is addressed by one github.run_id, killing GAP#6); job modules (needs: kernel — checks out modules@modules_ref and CRITICALLY clones the kernel patch source at kernel_ref not main, fixing GAP#4; consumes the kernel obj from this run's artifacts via needs); job compat (needs: toolchain); job assemble+boot (needs: kernel, modules, compat — checks out nextbsd@nextbsd_ref, builds mach.ko + libmach from THAT ref where the mux-drop lives, assembles the IMG consuming the three in-run artifacts via the override path NOT the latest-main scrape, runs boot-test.sh). NEVER publishes continuous, NEVER dispatches downstream. The run records the full {repo:ref} set in its inputs and posts a status check/comment back to every participating PR (integration/coupled => required check). Generalizes the ONE existing override seam (modules' run_id+fbsd_sha) from a single pinned upstream run to an arbitrary N-branch combination built fresh and coherent in a single recorded run.
Coupled change (syscall + multiplexor) flow: The coupled pair is exactly {nextbsd-kernel: syscall-limit} x {nextbsd: mux-drop} (the slot-limit patch lives in nextbsd-kernel/patches/; mach.ko + libmach both live in the nextbsd repo). Steps: (1) open PR #NN in nextbsd-kernel adding the lkmnosys-band-widening patch to patches/series; (2) open PR #MM in nextbsd dropping the mux: add wire_one(task_set_special_port/host_set_special_port/mach_port_move_member/event-bell) as dedicated sysents in mach_syscall_wire.c, delete the mach_trap_mux_sysent + sys_mach_trap_mux_trap switch in mach_traps.c, switch libmach/mach_traps.c to per-name resolve_syscall; (3) comment '/integrate kernel#NN nextbsd#MM' (the bot maps PR numbers to head refs and repository_dispatch's integration.yml with kernel_ref=<#NN head>, nextbsd_ref=<#MM head>, the other three blank => latest-green pin); (4) integration.yml builds toolchain (latest-green), cross-builds kernel@#NN (widened slot band -> kernel-obj), builds modules cloning kernel patch source AT #NN (GAP#4-correct), builds compat (latest-green base), assembles the IMG from nextbsd@#MM consuming THIS run's artifacts, boots it; (5) required green markers TASK-SPECIAL-PORT-OK (was mux op=2), HOST-BOOTSTRAP-OK (op=1), BOOTSTRAP-REMOTE-OK (depends on mach_port_move_member op=3 -> launchd), plus MACH-SMOKE-OK/LIBSYSTEM-KERNEL-OK/LIBXPC-OK; smoke-sysctls.sh additionally asserts each of mach.syscall.task_set_special_port/.host_set_special_port/.mach_port_move_member returns a DEDICATED >=0 slot and mach.syscall.mach_trap_mux is GONE; (6) the integration status check posts back to BOTH PRs as a required 'coupled-integration' check — neither can merge red. On green the merge queue merges them together: kernel#NN first (so its main artifact exists), then nextbsd#MM, re-running integration against the up-to-date target before each lands; on kernel#NN merge a pin-bump PR updates versions.lock to the new kernel artifact so subsequent nextbsd main builds ingest the coherent slot-raised kernel.
Today (VM): a kernel-only /integrate that actually boots drags in the nextbsd vmactions VM leg -> ~28m on-demand (run by label/comment to bound VM cost); if you only need 'does it compile + do modules wire', skip the boot leg for ~6m warm / ~10m cold. Native-cross: full boot-tested kernel PR ~6.5m warm / ~13.5m cold (same as B, generalized to N refs).
Today (VM): one /integrate run builds toolchain (reused tag ~0-2m) + kernel cross (~7m45s) + modules cross (~3m, parallel w/ compat) + compat (~3m) then the dominant vmactions VM leg (livecd ISO + mach.ko/libmach + QEMU boot) ~17m+3m => ~28m wall-clock, fully serialized on the single VM leg; run on-demand (label/comment) to bound cost. Native-cross: the whole coupled {kernel#NN, nextbsd#MM} boot-tested run drops to ~14.5m cold / ~7m warm and becomes cheap enough to run automatically on every cross-repo-labeled PR.
Ratings: coupling ease high speed medium infra simplicity medium blast-radius control high release flexibility high
Higher is better on every axis (green = high, amber = medium, red = low).
| Option | Coupling ease | Speed | Infra simplicity | Blast-radius control | Release flexibility |
|---|---|---|---|---|---|
| A Continuous releases on every repo + PRs consum | low | low | medium | high | high |
| B Artifact-override workflow_dispatch inputs | high | medium | high | high | medium |
| C1 Full monorepo | high | high | high | low | low |
| C2 Partial monorepo | high | high | medium | low | medium |
| D Hybrid integration meta-workflow over {repo:re | high | medium | medium | high | high |
Pick: NOW (today's VM world): adopt B (artifact-override inputs) as the coupling mechanism + the latest-green pin from A as the stability backbone. AFTER crossbuild lands: migrate to C2 (partial monorepo, toolchain stays an external pinned image) + a merge queue, keeping a thin D-style integration.yml for any ref combination that still spans a non-merged repo.
The syscall-limit + mux-drop pair must be built into ONE ISO and booted under QEMU before either merges — the failure mode (slot exhaustion -> sysctl -1 -> syscall(-1) faults) is invisible to compilation and to single-repo CI, so a boot-tested integration build is mandatory. Against that hard requirement: A NEVER delivers pre-merge coupled testing at any speed (it validates only after both halves are released), so it is the stability backbone, not the answer. B is the cheapest path to the goal and ~80% already built (modules already has run_id+fbsd_sha; toolchain already publishes :<branch> tags and gates the cascade on main), needs no standing infra, is idiomatic to these repos, and works TODAY in the VM world at ~28m on-demand — immediately unblocking the pair with one extra ISO dispatch. Its one hard prerequisite for THIS scenario is the GAP#4 fix (make the modules 'Apply patches to source' clone ref-aware), because the syscall-limit is a kernel-SOURCE change and without it mach.ko is KBI-desynced; that fix is small and is also required by D, so it is never wasted work. C1/C2 give the ideal ergonomics (one PR, one run, ~7m warm, GAP#4/#6/#7 eliminated by construction) but are HARD-BLOCKED until the IMG cross-builds on Linux, so they cannot be the now-answer. Once crossbuild lands, C2 is preferred over C1 because the toolchain genuinely rebuilds on a slower, source-bump-driven cadence and benefits from an immutable GHCR digest as the reproducibility anchor, and merging 4 histories is lower-risk than 5; fold the toolchain in later (-> C1) only if the external image boundary proves more annoying than valuable. D is the right generalization when 2-ref overrides stop sufficing or when a third coupled ref (e.g. a matching toolchain bump) appears, and it stays useful even after a monorepo for any combination spanning a still-split repo (toolchain/freebsd-src). Net cost-to-goal ranking: B (lowest, mostly built) < D (moderate, reuses B) < merge-queue/pin (low config) ; C highest one-time and blocked until crossbuild but best long-term steady-state. The pick threads these: B+pin now, C2+merge-queue (with residual D) later.
Generated from an 8-agent workflow (3 investigators: current topology + gaps, the real syscall/multiplexor change, coupled-change CI patterns; 4 option-designers: continuous-release, artifact-override, monorepo, hybrid integration + a shared time-to-test cost model; 1 synthesizer). Time-to-test figures derive from measured runs (nextbsd VM build ~17 min, kernel 3m09s warm / 7m45s cold, compat ~4 min, boot test ~3 min).