libdispatch Mach backend spike for freebsd-libxpc

Spike answering: do we really need to patch libdispatch to support DISPATCH_SOURCE_TYPE_MACH_RECV on FreeBSD? Who actually uses it? What happens if we don't? And could simpler per-consumer modifications work instead? Empirical grep over the local NextBSD, ravynOS, freebsd-launchd, and gershwin-developer trees plus a trade-off matrix for three implementation paths.

Contents

  1. Why libdispatch needs Mach support
  2. Who actually uses DISPATCH_SOURCE_TYPE_MACH_RECV
  3. What happens if we don't add Mach support
  4. Three options for adding it
  5. All in-scope apps — dispatch + Mach-dispatch requirements
  6. Install path: /System vs unix paths
  7. Recommendation
  8. References

1. Why libdispatch needs Mach support

libdispatch provides asynchronous event delivery through a uniform "source" abstraction: a consumer creates a dispatch_source_t for some event type (timer, fd-readable, signal, kqueue note, Mach message arrival), registers an event handler block, and the source delivers events on a target queue with no polling code in the consumer.

DISPATCH_SOURCE_TYPE_MACH_RECV is the source type for "a message arrived on this Mach port." Consumer code looks like:

port = mach_reply_port();
src = dispatch_source_create(DISPATCH_SOURCE_TYPE_MACH_RECV, port, 0, queue);
dispatch_source_set_event_handler(src, ^{
    /* called when a message arrives — drain it with mach_msg(MACH_RCV_MSG) */
});
dispatch_resume(src);

On Apple platforms libdispatch hooks this up via EVFILT_MACHPORT (a Darwin-only kevent filter) so the kernel signals the dispatch worker when a message hits the port. On FreeBSD, with HAVE_MACH off, the source type symbol doesn't exist at all — _dispatch_source_type_mach_recv is gated by #if HAVE_MACH in src/event/event_kevent.c:3249 with no fallback.

2. Who actually uses DISPATCH_SOURCE_TYPE_MACH_RECV

Earlier loose reading of the codebase suggested "only libxpc cares about this." Empirical grep across all local trees shows that's wrong:

ComponentFile / lineTypeWhat for
libxpc lib/libxpc/xpc_connection.c:214 MACH_RECV Async event delivery on a connection's local receive port
libnotify lib/libnotify/notify_client.c:355 MACH_RECV Notification reception via the global Mach notify port
notifyd usr.sbin/notifyd/notifyd.c:1230 MACH_RECV Daemon's server-port receive loop
notifyd usr.sbin/notifyd/notify_proc.c:352 MACH_SEND Dead-name and send-possible notifications on client ports
notifyutil usr.bin/notifyutil/notifyutil.c:374 MACH_RECV CLI tool's notification reception
SystemConfiguration framework configd/SystemConfiguration.fproj/SCNetworkConnection.c:2396 MACH_RECV Client subscribing to SC connection-state notifications
SystemConfiguration framework configd/SystemConfiguration.fproj/SCDNotifierInformViaCallback.c:610 MACH_RECV SCDynamicStore change callbacks via Mach

At least 5 distinct components in our porting scope use DISPATCH_SOURCE_TYPE_MACH_RECV: libxpc, libnotify, notifyd, notifyutil, SystemConfiguration framework. Each consumer that arrives later (mDNSResponder bridges, IPConfiguration helpers, future libxpc consumers) adds to this count. The expectation "Mach-receive sources Just Work" is baked into modern Apple-style C IPC code.

3. What happens if we don't add Mach support

With HAVE_MACH=0 on FreeBSD (the current default for swift-corelibs-libdispatch on non-Darwin), the symbol _dispatch_source_type_mach_recv simply does not exist in libdispatch.so. Consequences for each consumer:

ConsumerEffect at build timeEffect at runtime
libxpc FAILS to link — undefined reference to _dispatch_source_type_mach_recv library never produced
libnotify FAILS same way library never produced
notifyd / notifyutil FAILS daemon / CLI tool never produced
SystemConfiguration framework FAILSSCNetworkConnection and SCDNotifierInformViaCallback both reference the symbol framework can't be built — clients can't link -lSystemConfiguration
Apple launchd (clean launchd-842.92.1 import) unaffected — the 2014 source predates dispatch Mach sources; uses its own mach_msg loop in runtime.c fine
asl / syslogd / aslmanager / libasl unaffected — pure C, no libdispatch, no Mach fine

Net effect of doing nothing: libxpc, libnotify, notifyd, notifyutil, and the SystemConfiguration framework cannot be built at all. The Apple-source daemon ecosystem (other than the 2014 launchd and the legacy ASL daemons) stops being available.

That makes the libdispatch Mach work effectively load-bearing for the whole post-Phase-B roadmap, not just a libxpc concern.

4. Three options for adding it

Option 1 — Patch libdispatch with a new FreeBSD-Mach backend

Add a new src/event/event_mach_freebsd.c to swift-corelibs-libdispatch, gated on __FreeBSD__ && HAVE_LIBMACH. Defines _dispatch_source_type_mach_recv (and _send). Implementation: per-source polling thread that calls mach_msg_trap(MACH_RCV_MSG | MACH_RCV_TIMEOUT, timeout=small) and dispatches received messages to the source's target queue. Ship as a new patch in gershwin-developer/Library/Patches/ alongside the existing swift-corelibs-libdispatch.patch.

Code added~500-1000 lines in one new file
Code modified1 line in CMakeLists.txt
Consumers affected0 — consumers write vanilla dispatch_source_create(DISPATCH_SOURCE_TYPE_MACH_RECV, ...) code
Upstream coordinationOne patch to gershwin-developer; PR-able to swift-corelibs-libdispatch long-term
Cost when adding consumer N+1zero — new consumer just uses the existing source type

Pros: aligned with Apple's design (libxpc, libnotify, notifyd, SystemConfiguration framework code stays as-is); one place to fix bugs, optimize, profile. Cons: more total code than a single per-consumer patch; touches a shared library that other gershwin components also link.

Option 2 — Patch each consumer to use its own polling thread

Modify libxpc, libnotify, notifyd, notifyutil, and SystemConfiguration framework to skip dispatch_source_create(DISPATCH_SOURCE_TYPE_MACH_RECV, ...) and instead spawn a polling thread that calls mach_msg_trap and re-enqueues messages onto the consumer's target queue via dispatch_async. libdispatch stays vanilla.

Code added per consumer~100-200 lines for the polling thread + dispatch_async glue
Total across 5 consumers~500-1000 lines
Consumers affected5 — each gets its own polling thread implementation
Upstream coordination5 patches in 5 different projects; some sit in Apple-source-imports we re-pull on each version bump — rebase burden
Cost when adding consumer N+1+100-200 lines per new consumer — scales linearly with daemon count

Pros: libdispatch stays vanilla (no gershwin patch); each consumer is self-contained; works incrementally (port one daemon at a time). Cons: diverges from Apple's design in 5+ places; per-consumer thread overhead (each connection or each daemon has its own polling thread); patches to verbatim Apple imports complicate re-imports; scales linearly with new ports.

Option 3 — Build a small Mach-to-dispatch bridge library

Ship a new tiny library (e.g. libmach_dispatch.so) that exposes a clean API like:

typedef struct mach_dispatch_source *mach_dispatch_source_t;

mach_dispatch_source_t
mach_dispatch_source_create_recv(mach_port_t port, dispatch_queue_t queue,
    void (^handler)(mach_dispatch_source_t));

void mach_dispatch_source_resume(mach_dispatch_source_t src);
void mach_dispatch_source_cancel(mach_dispatch_source_t src);
void mach_dispatch_source_release(mach_dispatch_source_t src);

Implementation: polling thread that calls mach_msg_trap, dispatches via dispatch_async. libdispatch stays vanilla. Consumers (libxpc, libnotify, notifyd, etc.) substitute mach_dispatch_source_create_recv for dispatch_source_create(DISPATCH_SOURCE_TYPE_MACH_RECV, ...).

Code added in the bridge~300-500 lines (smaller than libdispatch backend since we own a simpler API)
Code modified per consumer~10-20 lines to switch API calls
Total across 5 consumers~50-100 lines of consumer changes + 300-500 lines of bridge
Consumers affected5 — small API substitution each
Upstream coordinationSelf-contained library in our repo + small patches to each consumer (still patches Apple-source imports)
Cost when adding consumer N+1~10-20 lines per new consumer

Pros: smaller bridge code than libdispatch patch (we own a simpler API); libdispatch stays vanilla; consumers don't have to know polling-thread details. Cons: still patches Apple-source imports (rebase burden); diverges from Apple's API names (mach_dispatch_source_create_recv vs dispatch_source_create); new consumers have to learn our bridge API.

Trade-off matrix

DimensionOpt 1 patch libdispatchOpt 2 per-consumer pollOpt 3 bridge library
Total new code (1st cut)~500-1000 lines, 1 file~500-1000 lines across 5 consumers~350-600 lines (bridge + consumer changes)
Apple-source-imports left untouchedyesno — patches verbatim Apple importsno — patches verbatim Apple imports
Rebase burden on Apple re-importslowhigh — 5+ patches to rebasemedium — 5+ small patches
Diverges from Apple's APInono (uses different internal mechanism, same Apple call site shape)yes — consumer uses our API names
Cost per future consumer~0+100-200 lines+10-20 lines
Polling thread count1 per dispatch source1 per consumer per source1 per bridge source
libdispatch stays vanillano — gershwin patchyesyes
Smoke-testable in isolationyeseach consumer separatelyyes

5. All in-scope apps — dispatch + Mach-dispatch requirements

Comprehensive matrix covering every component we plan to port (or have already ported) to FreeBSD as part of the broader Apple-source-services roadmap. Counts are from grep -rE "dispatch_[a-z_]*\(" over the local NextBSD, ravynOS, freebsd-launchd, and freebsd-launchd-mach trees; DiskArbitration counts are from spot-reads of apple-oss-distributions/DiskArbitration source on GitHub (no local clone yet).

ComponentPlain libdispatch useMach dispatch (SOURCE_TYPE_MACH_RECV/_SEND)Porting status
Libraries we build/ship
mach.ko (kernel) none — kernel can't link userland libdispatch n/a done — Phase B + Tier 1
libmach none — pure syscall wrappers n/a done — Phase C1
libdispatch itself (swift-corelibs-libdispatch + Mach backend patch) 3956 internal refs — this is the library required to provide the symbol — this spike's whole point Phase 0 — ship Mach backend as gershwin patch
libxpc moderate — 33 calls, 14 block literals, semaphores + queues + async 1 RECV in xpc_connection.c:214 Phase 2-4
liblaunch 10 calls — light (queue create + retain/release only) none existing legacy launch_data_t client; minimal forward port
libnotify 35 calls — moderate (2 sources, 13 blocks) 1 RECV in notify_client.c:355 Phase 7+ (after libxpc)
libasl 50 calls — moderate (1 source, 23 blocks) none — uses READ source type, not Mach Phase 7+
libSystemConfiguration (configd's client lib) 5 calls — light none directly — transitively via SystemConfiguration.framework Phase 6
SystemConfiguration.framework (configd's API surface) heavy — 124 calls, 60 blocks, 17 semaphores, 4 sources 2 RECV (SCNetworkConnection.c:2396, SCDNotifierInformViaCallback.c:610) Phase 6
DiskArbitration.framework moderate — DASessionSetDispatchQueue is core API; dual-mode CFRunLoop / dispatch-queue at least 1 RECV in DASession.c (from DASessionSetDispatchQueue impl on GitHub) Phase 8+ — deferred; may keep a sockets-based alternative implementation that sidesteps Mach IPC entirely
Daemons
Apple launchd-842.92.1 (clean import for Phase 5) ~10 real calls after filtering (dispatch_main, dispatch_once, 1 PROC source) none — 2014 source predates dispatch Mach migration; uses its own mach_msg loop Phase 5
freebsd-launchd minimal rewrite (current shipping) 27 calls — SIGNAL + READ + TIMER sources, dispatch_main none — AF_UNIX IPC, not Mach shipping today
launchctl none none Phase 5 (with launchd)
asl (Apple syslogd-replacement daemon) heavy — 145 calls, 16 sources (TIMER/READ/SIGNAL/VNODE), 66 blocks none — uses TIMER/READ/SIGNAL/VNODE source types only Phase 7+
syslogd (FreeBSD native) none — pure FreeBSD, no dispatch none n/a — we're replacing this with Apple's asl-aware version
aslmanager 4 calls — light (queue + dispatch_main) none Phase 7+ (with asl)
notifyd heavy — 114 calls, 14 sources (TIMER/SIGNAL/DATA/VNODE/PROC), 31 blocks 1 RECV (notifyd.c:1230) + 1 SEND (notify_proc.c:352) Phase 7+
notifyutil (CLI tool) light 1 RECV (notifyutil.c:374) Phase 7+ (with notifyd)
configd daemon (configd.tproj) 4 calls — light direct use (SIGNAL + dispatch_main); heavy through the framework none directly — works through SystemConfiguration.framework Phase 6
configd Plugins (IPMonitor, KernelEventMonitor, etc.) 54 calls — moderate (TIMER + READ sources, blocks) none directly Phase 6 (partial — IPMonitor's NWNetworkAgentRegistration blocker per the Foundation spike)
scutil CLI 7 calls — light none Phase 6
diskarbitrationd heavy — uses dispatch_mach_create_f, dispatch_mach_connect, IONotificationPortSetDispatchQueue, xpc_set_event_stream_handler uses dispatch_mach_t channels — even higher-level than MACH_RECV sources; requires the DISPATCH_MACH_SPI private API Phase 8+ — deferred; possibly replaced with a sockets-based daemon that sidesteps Mach entirely
DiskArbitrationAgent (per-user GUI) likely heavy (Apple Cocoa app) likely yes (uses DASessionSetDispatchQueue from the framework) Phase 8+; needs GNUstep Foundation/AppKit, separate concern
mDNSResponder daemon core (mDNSCore/, mDNSPosix/) moderate-to-heavy in mDNSMacOSX/ (Apple-only); pure POSIX in mDNSPosix/ likely some in mDNSMacOSX/; none in mDNSPosix/ Phase 7+ — we'd port the mDNSPosix/ path, not mDNSMacOSX/
IPConfiguration daemon (Apple's bootp client) heavy — timer.c + FDSet.c ~494 LoC of dispatch usage per the IPConfiguration porting plan uncertain — per the plan, MIG/Mach IPC heavy; on modern Apple the server.c uses XPC which sits on dispatch+Mach Phase 8+ — deferred; current ISO uses dhcpcd from FreeBSD ports
Future / optional
libxpc bootstrap server (Phase 3) moderate — would use dispatch sources for the bootstrap port required — bootstrap port is exactly the kind of long-lived Mach receive that needs dispatch Phase 3
libCoreFoundation (swift-corelibs CF) supplementary system library some — for CFRunLoop version1 sources if we route through dispatch yes — CFMachPort and CFRunLoop v1 sources are Mach-backed Phase 5.5 (now driven by launchctl-corefoundation-spike; replaces the older libCFRuntime proposal)
NSXPCConnection (downstream GNUstep contribution) moderate — wraps libxpc, inherits its dispatch use transitively via libxpc Optional / downstream / not on critical path

Summary by impact

CategoryComponentsImplication for Phase 0
Need Mach dispatch (MACH_RECV or higher) libxpc, libnotify, notifyd, notifyutil, SystemConfiguration framework, DiskArbitration framework (if we use Apple-shape impl), diskarbitrationd (if Apple-shape), libxpc bootstrap server Phase 0 blocks all of these. ~8 components.
Use libdispatch but NOT Mach dispatch Apple launchd-842 (light), freebsd-launchd rewrite, asl (heavy with TIMER/READ/SIGNAL/VNODE), aslmanager, configd daemon, configd plugins, scutil, mDNSResponder mDNSPosix, IPConfiguration (timer/FD), libasl, libSystemConfiguration, liblaunch Phase 0 unblocks the Mach RECV symbol but these components don't need it. They use plain libdispatch which builds fine on FreeBSD today (subject to the gershwin kqueue patch).
No libdispatch at all mach.ko, libmach, launchctl, syslogd (FreeBSD native) Unaffected by the Phase 0 / libdispatch path entirely.

Caveats and alternatives

6. Install path, vendoring, and build integration

Decided (per install layout spike §4): libdispatch installs as /usr/lib/system/libsystem_dispatch.so with Apple-canonical naming, alongside libsystem_kernel, libsystem_xpc, libsystem_launch, and libsystem_blocks. Headers at the standard /usr/include/{dispatch,os}/*.h. Build pipeline: vendored under freebsd-launchd-mach/src/libdispatch/ and built in our build.sh chroot, not via gershwin-developer.

Earlier revisions of this spike weighed four candidate paths (/System/Library/Libraries/, /usr/lib/, /usr/local/lib/, and the layout-spike's later addition of /usr/lib/system/). The Libsystem-prefix path won on all the criteria that mattered: Apple-canonical naming, coherent grouping with the rest of the libsystem_* family, isolated namespace away from FreeBSD base proper, no pkgbase-collision risk for the already-shipped libBlocksRuntime, and clean ldconfig setup via a single drop-in.

6.1. Source vendoring

swift-corelibs-libdispatch is committed verbatim under freebsd-launchd-mach/src/libdispatch/. Two patches are applied as commits on top of the upstream source in our tree:

  1. FreeBSD performance / correctness patch (gershwin-developer's existing swift-corelibs-libdispatch.patch) — see §6.4. Applied first.
  2. Mach backend (this spike's deliverable) — new src/event/event_mach_freebsd.c + minimal CMakeLists.txt wiring, gated on __FreeBSD__ & HAVE_LIBMACH. Links against -lsystem_kernel from our libmach build (layout spike §3).

Both patches travel as ordinary commits in our git history (no separate .patch file in Library/Patches/ — the gershwin patch-script workflow is retired for libdispatch in this project; gershwin-developer can either consume our vendored tree or keep its own copy). Vendoring trades repo size for hermetic builds, no network at build time, and audit-friendly diffs.

6.2. Block.h coordination — we ship it now

Reversing the prior recommendation. Earlier revisions said "defer to FreeBSD pkgbase's FreeBSD-libblocksruntime-15.0 for /usr/include/Block.h and /usr/lib/libBlocksRuntime.so." The new direction (per install layout spike §15):

Net effect: Block.h on the system comes from our build, not pkgbase. Single source of truth. No collision because pkgbase isn't installed. The "single Block.h on the system" property the prior recommendation valued is preserved — we just own the source rather than borrowing from pkgbase.

6.3. Build integration in build.sh

libdispatch builds as a new step 3d in freebsd-launchd-mach/build.sh, immediately after libmach (libsystem_kernel) installs. Outline:

  1. rsync src/libdispatch -> chroot:/tmp/libdispatch (chroot stays git-free).
  2. chroot $WORK/rootfs cmake with -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=lib/system -DINSTALL_BLOCK_HEADERS_DIR=/usr/include -DINSTALL_DISPATCH_HEADERS_DIR=/usr/include/dispatch -DINSTALL_OS_HEADERS_DIR=/usr/include/os -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Release — uses cmake/ninja already in buildpkgs.txt and clang already in buildpkgs-base.txt.
  3. Built artifacts land at /usr/lib/system/libsystem_dispatch.so + sonname symlink, /usr/include/dispatch/*.h, /usr/include/os/*.h, /usr/include/Block.h.
  4. Verification asserts (mirroring libmach's): file existence, sonname symlink correct, ldd of the test binary resolves libsystem_dispatch.so.0 into /usr/lib/system/.

ldconfig hint reuses the existing drop-in at /usr/local/libdata/ldconfig/freebsd-launchd-mach (added for libmach — it lists /usr/lib/system). No new ldconfig setup needed; the runtime linker already finds the directory.

6.4. Required FreeBSD performance / correctness patch

swift-corelibs-libdispatch needs gershwin-developer/Library/Patches/swift-corelibs-libdispatch.patch applied for FreeBSD to perform correctly. Hunks below; all are FreeBSD-kqueue accommodations and have nothing to do with Mach:

HunkFileFixWhy it matters
1 src/event/event_kevent.c FreeBSD-specific timer fflags init macros that exclude NOTE_ABSOLUTE on FreeBSD (other systems use it) FreeBSD's kqueue NOTE_* constants differ from Apple's; without the patch, timers either don't compile or use wrong semantics.
2 src/event/event_kevent.c Replace zero-timeout kevent() polling with a 1ms minimum delay on non-QOS systems (FreeBSD/Linux) Without this, libdispatch worker threads tight-spin on kevent() with no events, burning CPU.
3 src/event/event_kevent.c Enforce a minimum timer delay (default 10ms, configurable via LIBDISPATCH_TIMER_MIN_DELAY_MS env var) to prevent scheduling timers at or before now Without this, timer programming triggers immediate re-arms and tight kevent loops on FreeBSD's kqueue.
4 src/event/workqueue.c Fix a loop-variable typo: for (int j = 0; i < count; ++i)j < count; ++j in _dispatch_workq_count_runnable_workers() Pre-existing bug in libdispatch's FreeBSD workqueue thread-counting; would scan past array bounds. Pure correctness fix.

Applied via the existing gershwin-developer/Library/Patches/apply_swift-corelibs-libdispatch_patch.sh at vendoring time (one-shot), then committed into our tree as part of the initial vendor. Subsequent maintenance: rebase onto upstream when bumping the libdispatch version.

6.5. Live-reinstall semantics under our chroot model

The earlier "/System live-reinstall fragility" discussion is no longer load-bearing because libdispatch ships inside the rootfs.uzip on the live ISO — consumers see the file via the in-kernel unionfs, and it's installed once at build time, not interactively. Wholesale-replace concerns from the gershwin /System path don't apply.

For an installed (non-live) FreeBSD where freebsd-launchd-mach is the system source: pkg upgrade handles /usr/lib/system/libsystem_dispatch.so atomically (pkg replaces files via a temp-then-rename pattern). Already-running processes keep their mapped libdispatch; new processes pick up the upgraded version. Standard FreeBSD pkg semantics, no atomic-rename ceremony needed.

6.6. Recommendation

Recommended (and decided):

7. Recommendation

Option 1 — patch libdispatch. The dominant factor is "every Apple-source consumer expects DISPATCH_SOURCE_TYPE_MACH_RECV to exist." Option 1 honors that expectation; Options 2 and 3 force every Apple-source import to be modified, which complicates re-imports forever.

The earlier mental model "only libxpc cares" understated the scope. With 5+ consumers in scope already (libxpc, libnotify, notifyd, notifyutil, SystemConfiguration framework) and more arriving as we port additional daemons, the per-consumer linear cost of Options 2 and 3 is real. Option 1's higher upfront cost (~500-1000 lines of one new file in a patch) amortizes across every present and future consumer.

Secondary factor: Apple-import rebase burden. Our project's overall pattern is "import Apple source verbatim, patch for FreeBSD adaptation as a separate layer." Options 2 and 3 force patches into the verbatim Apple files themselves; every time we re-pull from apple-oss-distributions/<daemon>, the patches need rebasing. Option 1 isolates the FreeBSD-specific work in gershwin-developer/Library/Patches/ — the same place gershwin already maintains a libdispatch patch.

Tertiary factor: consistency with the broader project pattern. The libxpc plan's Phase 0 already specifies a libdispatch Mach backend patch in gershwin-developer; this spike confirms that choice was right and tightens its scope.

What stays unchanged from the libxpc plan

What this spike adds to the libxpc plan

6. References

Local trees grep'd

grep -rn "DISPATCH_SOURCE_TYPE_MACH_RECV\|DISPATCH_SOURCE_TYPE_MACH_SEND" \
    nextbsd/ ravynos/ freebsd-launchd/ gershwin-developer/

Related plans

Last updated 2026-05-12. Spike compiled from empirical grep over the local NextBSD, ravynOS, freebsd-launchd, and gershwin-developer trees plus prior research at freebsd-libxpc-plan §6. The 5-consumer count for DISPATCH_SOURCE_TYPE_MACH_RECV is reproducible: grep the trees, count distinct source files, get the same 7 unique sites. Be aware that more consumers will be added as we port mDNSResponder, IPConfiguration, and additional Apple-source daemons; the cost arguments for Option 1 strengthen with each.