One tree to rule them all: consolidating NextBSD into a single freebsd-src monorepo

0What you actually have today

Grounding the plan in the recon agent's reading of your live checkouts and the nextbsd-redux / pkgdemon orgs — including two facts that change the plan.

The current architecture is ~10 repos wired into a cascading repository_dispatch pipeline. freebsd-src (a publish-only mirror fork, ~3.36 GB) syncs daily on releng/15.1, which fans out to a cross-toolchain build, then kernel → kernel-modules on one path and freebsd-compat (buildworld base) → userland → pkg → ISO on the other. Each repo carries its own .github/workflows; PR runs are build-only and non-cascading.

The through-line: the monorepo isn't inventing your dependency graph — it's re-encoding the cascade you already hand-wired across repos as ordinary in-tree build ordering plus (optionally) BuildKit stage edges, where cache invalidation is computed from content digests instead of repository_dispatch plumbing.

1The sync model — and why merge, not patches

Your instinct ("changes as code on the tree, sync often via a PR, not patch files, not an in-network fork") is correct. Here is the exact model and the evidence.

2Phase 1 — stand up pkgdemon/nextbsd

Concrete steps: create the empty repo, seed it from a snapshot of FreeBSD main, wire the upstream remote, and land the sync-via-PR workflow. Nothing here touches your existing repos — it's additive and reversible.

2.1 — Seed the tree (single branch, no deep history)

Importing FreeBSD's full history (~2.6 GB) would put you near GitHub's 5 GB discomfort line before your first commit and tax every CI clone forever. Take a snapshot instead:

# 1. Create the empty repo first (gh or the web UI):
gh repo create pkgdemon/nextbsd --private --description "NextBSD monorepo"

# 2. Grab main only, shallow, into a scratch dir
git clone --branch main --single-branch --depth 1 \
    https://github.com/freebsd/freebsd-src.git seed
cd seed

# 3. Re-root history at this snapshot: first commit = "Import FreeBSD main @HASH"
UP=$(git rev-parse HEAD)
rm -rf .git
git init -b main
git add -A
git commit -m "Import FreeBSD-src main @ ${UP:0:12}"

# 4. Push to your monorepo
git remote add origin git@github.com:pkgdemon/nextbsd.git
git push -u origin main

# 5. Add upstream for future syncs
git remote add freebsd https://github.com/freebsd/freebsd-src.git
git config remote.freebsd.tagOpt --no-tags
git config rerere.enabled true          # remember conflict resolutions
git fetch freebsd main

Working-tree size lands ~1–1.5 GB — comfortably inside GitHub's recommended range and fast to clone in CI. It grows only as you add code plus the deltas of upstream commits you actually merge.

2.2 — The sync-via-PR workflow

# .github/workflows/upstream-sync.yml
name: Upstream FreeBSD sync
on:
  schedule: [{ cron: '0 6 * * 1' }]     # Mondays 06:00 UTC
  workflow_dispatch: {}                  # the "click to sync" button you asked for
permissions: { contents: write, pull-requests: write }
jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - name: Merge upstream (allow conflicts)
        id: merge
        run: |
          git config user.name  "nextbsd-sync[bot]"
          git config user.email "sync@nextbsd.invalid"
          git remote add freebsd https://github.com/freebsd/freebsd-src.git
          git fetch --no-tags freebsd main
          BR="sync/$(date +%Y-%m-%d)"; echo "branch=$BR" >>"$GITHUB_OUTPUT"
          git switch -c "$BR"
          if git merge --no-ff --no-edit freebsd/main; then
            echo "conflicts=false" >>"$GITHUB_OUTPUT"
          else
            git add -A
            git commit --no-edit -m "MERGE CONFLICTS - resolve before merging"
            echo "conflicts=true" >>"$GITHUB_OUTPUT"
          fi
          git push -u origin "$BR"
      - name: Open PR
        env: { GH_TOKEN: "${{ secrets.GITHUB_TOKEN }}" }
        run: |
          gh pr create --base main --head "${{ steps.merge.outputs.branch }}" \
            --title "Sync upstream FreeBSD main ($(date +%F))" \
            --body "Automated merge. Conflicts: ${{ steps.merge.outputs.conflicts }}. Merge with a MERGE COMMIT — never squash."

• Merge the sync PR with a real merge commit — squashing collapses upstream's individual commits and poisons the next merge's base (phantom conflicts).
• Never force-push main — history is append-only; that's the whole point of choosing merge over rebase.
• Enable Settings → Actions → General → "Allow GitHub Actions to create and approve pull requests." If an upstream merge ever touches .github/workflows/**, the default GITHUB_TOKEN can't push it — use a PAT with workflow scope.
• Resolve conflicts in your local clone (where rerere is on) after checking out the sync branch — CI can't easily share the rr-cache.

3Tree layout & the one ownership rule

Clean upstream merges and obvious "what's ours" are the same goal, served by one rule.

When you must modify an upstream file in place, wrap it: /* NEXTBSD-BEGIN: reason */ … /* NEXTBSD-END */. Then grep -rn NEXTBSD enumerates your entire in-place footprint — your ownership map and conflict-risk register in one command. Wire new subdirs in with the smallest possible edit (one line in a SUBDIR list). Never reformat or relocate upstream files "while you're in there" — churn manufactures conflicts against every future upstream commit.

Ownership becomes trivially answerable: under a next* path → ours; grep NEXTBSD hits it → ours-modified-upstream; otherwise → pristine upstream.

4Phase 2 — Mach, unionfs, and the kld/kext decision

"Integrate mach & unionfs patches as code in the branch, not as patch files." Here's what becomes an in-tree patch vs a loadable module — and the verdict on kexts.

4.1 — Where Mach and unionfs actually live

4.2 — Where IOKit earns its keep (narrowly)

A source-compatible IOKit framework (vendored libkern/OSObject/IOService/IORegistry compiled into .kos) is feasible but expensive — a multi-engineer-quarter effort per family, buying source compatibility, never binary. Rule of thumb: emulate IOKit only where you're reusing IOKit-shaped Darwin source. Filesystems, Mach glue, and most drivers should be native kld. GPU/graphics is the one place IOKit source reuse (IOGraphicsFamily/accelerators) can genuinely pay off; otherwise use drm-kmod. HID/audio: native evdev/sound(4) is usually less work than IOHID/IOAudio.

Document loading Apple binary kexts as an explicit non-goal. Your existing ko2kext.sh + personality generators produce kext packaging/metadata around FreeBSD .kos — that's the pragmatic middle ground and stays; it is not the same as running Apple's binaries.

5Userland into buildworld — and dropping cmake

The sanctioned hooks, the CMake→bsd.mk translation, and the decision rule for "in world vs a package."

5.1 — The LOCAL_* hooks (the official answer)

You do not patch base tree Makefiles. build(7) ships three hooks; set them once in /etc/src.conf and they apply to every buildworld/installworld/buildenv/cross-build/make release:

# /etc/src.conf
LOCAL_LIB_DIRS= nextbsd/lib/libCoreFoundation nextbsd/lib/libmach nextbsd/lib/libasl
LOCAL_DIRS=     nextbsd/usr.sbin/launchd nextbsd/usr.sbin/configd nextbsd/usr.bin/...
LOCAL_MODULES=  nextunionfs nextmach          # dirs under sys/modules

• LOCAL_LIB_DIRS builds during the libraries phase (early), so your programs can link them.
• LOCAL_DIRS builds during the everything phase.
• LOCAL_MODULES builds/installs as part of buildkernel/installkernel.
• There is no magic /usr/src/local auto-discovery — the mechanism is explicit. Keep everything under one nextbsd/ subtree pointed at by the hooks.

5.2 — Minimal Makefiles & the CMake translation

A typical program Makefile is 3–10 lines. Objective-C/GNUstep components compile .m fine but need explicit flags (-fobjc-runtime=gnustep-2.0 -fblocks, GNUstep header/lib paths) and usually WARNS?=0 for imported code. This works only if the GNUstep/libobjc2 stack they link is present — per your stack-provenance notes it's a source build into /System/Library/Libraries, so either give those their own LOCAL_LIB_DIRS entries or keep them as a package the world links against.

6PKGBASE shape — compose, don't strip

Your two mid-flight points: think about pkg base shape changes, and — you don't need to drop init or other unused base bits from the tree; let them ride in FreeBSD-runtime and compose ISOs from a minimal pkgbase set plus your unique NextBSD packages. Confirmed correct — with one nuance that's per-component.

6.1 — How pkgbase slices base

make packages (a.k.a. packagesystem) reads the METALOG that installworld/installkernel emit — every file stamped with a package= tag (set declaratively in Makefiles via PACKAGE=, defaulting to utilities) — groups files by tag, and emits one .pkg per group via pkg create. Per-package metadata comes from UCL templates in src/release/packages/*.ucl (rendered by generate-ucl.lua). It's the same pkg(8) as ports, just a separate FreeBSD-*-namespaced repo (base_latest). A full base is ~200–290 packages.

6.2 — Per-component: does "just don't install it" work?

Rule: reserve WITHOUT_* for genuine build-cost wins (toolchain, tests, lib32, debug — exactly what your compat flags already do), not for curation. Curation is a compose-time pkg install decision. This is the pkgbase realization of §5's in-base-vs-package rule.

6.3 — Minimal ISO set + derivative shape changes

Minimal bootable set to compose with your NextBSD-* packages: FreeBSD-clibs + FreeBSD-runtime + FreeBSD-kernel-generic (+ FreeBSD-rc only if you keep rc) + a few utilities (FreeBSD-utilities, FreeBSD-caroot/certctl for TLS, pkg itself). vermaden booted a full pkgbase system in 300 MB RAM; the minimal set is a fraction of the 208-package base. The build flow — bootstrap pkg into an empty root, pkg -r $ROOTFS install FreeBSD-clibs FreeBSD-runtime FreeBSD-kernel-generic … NextBSD-everything, regenerate /etc DBs, purge cache, makefs — is exactly what your builders already do; only the install line's contents change.

• Tag your in-tree tools: add PACKAGE=nextbsd-<component> to each LOCAL_DIRS Makefile so make packages emits clean NextBSD-* packages instead of dumping into FreeBSD-utilities.
• Metapackages: keep NextBSD-everything; add a slimmer NextBSD-base for headless/minimal images so "the ISO set" is a one-liner.
• Keep base named FreeBSD-*, layer NextBSD-* on top. Renaming all base packages is churn you'd re-merge forever, and the FreeBSD-* vs NextBSD-* split cleanly mirrors your existing compat-vs-userland boundary. Fork a package's name to NextBSD-* only when you've genuinely patched its contents (e.g. a modified runtime/init), so version comparison never pulls upstream's over yours. This is exactly the GhostBSD/TrueOS-lineage pattern.
• Repo authority: keep your REPOS_DIR isolation (you already point the client at only the NextBSD flat repo) and set explicit priority in the shipped NextBSD.conf.

Uncertain and worth a grep -rl 'PACKAGE=' /usr/src on your 15.x checkout: the exact tag for init and newsyslog, and the precise per-branch membership of the individually-packaged daemons.

7Replacing Apple syslog/ASL with FreeBSD's

Can stock FreeBSD syslog replace Apple's ASL, and can ASL callers work against it? Yes — feasibly and at low risk.

Do not port Apple's syslogd wholesale (the Darling route) — it drags in libnotify/notifyd, launchd socket-activation, and the ASL DB manager to duplicate a daemon FreeBSD already ships. Going to FreeBSD syslogd severs those couplings, which is desirable: no notify coupling (and your IPC is DO-based anyway), no launchd socket handoff (syslogd opens /var/run/log itself under rc), newsyslog(8) replaces aslmanager rotation. The only lost capability is structured log querying (a Console.app-class browser) — which NextBSD doesn't ship. Explicitly exclude modern os_log/Unified Logging; your ASL-era sources predate it and it's a heavy binary-format rabbit hole. This lets you delete the entire src/syslog/ Apple tree from the userland, shrinking maintenance and licensing surface.

8Selective & incremental builds

Only rebuild the kernel, only rebuild world, and make both fast — the real flag names.

7.1 — Partial builds

# kernel only
make buildkernel   KERNCONF=NEXTBSD          # NO_MODULES=yes / MODULES_OVERRIDE="a b" to scope modules
make installkernel KERNCONF=NEXTBSD

# world only
make buildworld

# one component the cross-safe way (after a world):
make buildenv TARGET=amd64 TARGET_ARCH=amd64  # spawns a shell with the tree's toolchain
  cd nextbsd/usr.bin/nbtool && make obj && make && make install DESTDIR=/staging

# rebuild just one subtree but keep libs/includes current
make buildworld SUBDIR_OVERRIDE=nextbsd/usr.bin/nbtool -DNO_CLEAN

7.2 — Fast incremental: meta mode + ccache

7.3 — Cross-building on Linux

Verdict: world + kernel cross-build from Linux is officially supported and CI-tested via tools/build/make.py (which bootstraps bmake + the tree's tools). Host needs bmake libarchive-dev clang-NN lld-NN. Building release media (ISO/memstick) on Linux is NOT supported — make release needs a FreeBSD host. This is precisely your intended split: continuous cross-builds on Linux, real releases on a NextBSD box.

env MAKEOBJDIRPREFIX=/obj ./tools/build/make.py --bootstrap-toolchain \
  TARGET=amd64 TARGET_ARCH=amd64 buildworld buildkernel KERNCONF=NEXTBSD -j$(nproc)

9Containerized layer caching

Your added idea — container + caching to speed obj at different layers — is feasible and a cleaner encoding of the artifact cascade you already run. BuildKit computes the invalidation you currently hand-wire.

Two mechanisms, don't conflate them: image layers (content-digest keyed, shareable, push to a registry) and cache mounts (RUN --mount=type=cache,target=/usr/obj — persistent scratch, builder-local, not part of any layer). Map the tiers to multi-stage FROMs. The subtlety that bites everyone: the FreeBSD source tree is one directory — a single COPY src/ /usr/src means any edit busts every downstream stage. Split the COPY by subtree so a sys/-only change doesn't invalidate the world-obj layer. (World legitimately reads some sys/ headers, so partitioning is imperfect — you accept some over-rebuild at the world/kernel boundary.)

10CI/CD topology & "patch checks"

One orchestrator detects what changed and calls reusable per-artifact workflows conditionally. Keep two axes separate: selective (PR/push, fast, no ISO) vs full/release (manual or nightly, everything).

9.1 — Selective builds

Use a detect-changes job with dorny/paths-filter emitting boolean outputs consumed downstream via needs + if (workflow-level paths: can't do per-job granularity). Filter sys/**→kernel, bin lib usr.bin usr.sbin contrib share/mk→world, ports nextbsd-pkg→pkg. Two gotchas: put share/mk/** and toolchain paths in the broadest filter (a Makefile change must force a full rebuild); and because a skipped upstream makes needs see result == 'skipped' (falsy), use if: always() && … and check .result explicitly, or the job silently never runs.

9.2 — "Patch checks" on contribution

Three tiers of PR gate scoped to what changed:

• Tier 1 — style/lint (always, seconds): FreeBSD ships tools/build/checkstyle9.pl in-tree (mechanical style(9)); run it against only changed files from git diff --name-only. Add git clang-format --diff and mandoc -Tlint/igor for man pages.
• Tier 2 — build-only-affected: compile just the changed component (a changed sys/dev/foo/ → targeted sys/modules/foo build) against a warm obj tree — minutes, not hours.
• Tier 3 — required status checks: mark lint required in branch protection. Because path-filtered jobs get skipped, use the always-run-detect + conditional-build pattern (a required check that never reports is a merge deadlock).

9.3 — Where each artifact runs

9.4 — Keep it DRY

Replace the N-repo duplicated pipelines with reusable workflows (on: workflow_call) per artifact + composite actions for shared setup (checkout + ccache restore + freebsd-vm/self-hosted select). Decision logic (path filters, if) lives only in the orchestrators; how-to-build lives only in wf-*.

.github/
  workflows/
    ci.yml          # orchestrator: detect-changes + conditional per-component (PR/push)
    continuous.yml  # nightly/on-green full build → artifacts + snapshot pkgs
    release.yml     # workflow_dispatch: branch+tag → full pipeline → gh-release
    upstream-sync.yml
    wf-lint.yml  wf-kernel.yml  wf-world.yml  wf-packages.yml  wf-iso.yml   # reusable
  actions/
    setup-freebsd/  ccache/     # composite

11Packages, ISOs & releases

Continuous vs tagged, and the 2 GiB wall you've already hit.

• Continuous (nightly + on green main): full cross-build → upload-artifact (kernel/world tarballs; 2 GiB/artifact cap) + push snapshot pkgs to nextbsd-pkg. No tag, short retention.
• Tagged release (workflow_dispatch, manual): operator triggers release.yml with a version → creates the release branch + tag → full world→kernel→ISO on self-hosted/real hardware → softprops/action-gh-release attaches the ISO + checksums. This is exactly your "build real releases myself on a real NextBSD kernel, push continuous ones from Linux crossbuild" split.

12Suggested sequencing & open decisions

Open decisions — flagged for breakout / sidebar

Other open decisions

• Snapshot vs stable branch to track. Your current mirror tracks releng/15.1, not main. Tracking main gets you the newest tree (and the best tools/build/make.py cross support) but more churn; tracking a releng/stable branch is calmer and is GhostBSD's choice. The sync model is identical either way — just point the freebsd remote at the branch you pick.
• Do the old component repos become archives or subtrees? Cleanest is to archive them read-only once their code lands in the monorepo, keeping the git history reachable but frozen.
• How aggressively to containerize. The BuildKit layer model is a real upgrade but adds Dockerfile + cache-key discipline as new maintenance surface. A valid lighter path: skip containers initially, run selective builds on a self-hosted FreeBSD runner with persistent obj/ccache, and add BuildKit layering only if cross-runner sharing becomes the bottleneck.

Every load-bearing claim above traces to a source-cited research pass (FreeBSD build(7)/src.conf(5)/release(7), HardenedBSD's migration, Apple/Darling ASL & IOKit sources, GitHub Actions & BuildKit docs). Two FreeBSD flag spellings (WITHOUT_CROSS_COMPILER, CROSS_TOOLCHAIN) are flagged moderate-confidence — verify against your tree before quoting. No repositories were created and no code was written in producing this document.