← Back · companion to Dropping libxpc and libdispatch

The Inverted Run Loop

NextBSD builds SCDynamicStoreCreateRunLoopSource on top of libdispatch. Apple builds it on CFMachPort and has since before libdispatch existed. That inversion is measurable — and it turned out not to be what crashes.

Measured 2026-09-01 on arm64 hardware, against Apple’s published configd sources, and across all 200 CI boot logs. Root cause identified — two independent surveys converged on the same code.

1. When SCNotify started using libdispatch

Apple’s SCDNotifierInformViaCallback.c is the file NextBSD’s SCNotify.c is a port of. Two consecutive releases settle the question:

SCDNotifierInformViaCallback.c	CFMachPort	CFRunLoopSource	dispatch_
configd-212.2 — 10.5 Leopard	17	7	0
configd-289 — 10.6 Snow Leopard	22	7	23

SCDynamicStoreSetDispatchQueue is defined in 289 and absent in 212.2. So SCNotify adopted libdispatch in Mac OS X 10.6 — the same release that introduced GCD. It has been dispatch-aware from the first day dispatch existed.

This is consistent with how that code is shaped generally. The 10.5 tree already shipped four independent delivery mechanisms as four separate files — SCDNotifierInformViaCallback.c, SCDNotifierInformViaFD.c, SCDNotifierInformViaMachPort.c, SCDNotifierInformViaSignal.c. Adding a fifth was the established pattern, not a migration.

2. The inversion, stated plainly

In Apple’s stack the dependency runs one way:

Apple
  SCDynamicStoreCreateRunLoopSource  ──>  CFMachPort        (older, dispatch-free)
  SCDynamicStoreSetDispatchQueue     ──>  libdispatch       (newer, added 10.6)
                                          two independent paths

In this tree it runs the other way:

NextBSD
  SCDynamicStoreSetDispatchQueue     ──>  libdispatch       (iter 2, built first)
  SCDynamicStoreCreateRunLoopSource  ──>  libdispatch       (iter 3, built ON the above)
                                          one path, stacked

SCNotify.c states the reason in its own header comment:

“Apple also offers SCDynamicStoreCreateRunLoopSource, built on CFMachPort. This repo’s libCoreFoundation does not compile CFMachPort (it is gated to TARGET_OS_MAC), so the run-loop-source variant is a later iteration.”

Concretely: the dispatch path takes the caller’s queue, while the run-loop path has no caller queue and so __sc_notify_start spins up a private serial one:

if (storePrivate->notifyStatus == Using_NotifierInformViaDispatch) {
        q = storePrivate->dispatchQueue;
} else {
        storePrivate->notifyQueue =
            dispatch_queue_create("com.apple.SCDynamicStore.notify", NULL);
        q = storePrivate->notifyQueue;
}

The run-loop path therefore carries strictly more machinery here than it does upstream, where it carries less. That is the whole finding, and it predicts which of the two tests fails.

3. What actually crashes

Measured on arm64 hardware against the CI main userland: scrltest segfaults in 13 of 30 runs. It passes under lldb, because the debugger’s timing hides the race — so it had to be stressed bare and caught with a core.

thread #1, name = 'DispatchWorker', stop reason = signal SIGSEGV
  frame #0: libsystem_dispatch.so`_dispatch_kevent_merge + 28
  frame #1: libsystem_dispatch.so`_dispatch_kq_drain + 144
  frame #2: libsystem_dispatch.so`_dispatch_mgr_invoke + 144
  frame #3: libsystem_dispatch.so`_dispatch_mgr_thread + 164

And on another thread, libdispatch’s own kevent error reporter, reached from knote teardown:

thread #3, name = 'DispatchWorker'
  frame #7:  libsystem_dispatch.so`_dispatch_bug_kevent_client + 436
  frame #8:  libsystem_dispatch.so`_dispatch_kq_drain + 244
  frame #9:  libsystem_dispatch.so`_dispatch_kq_unote_update + 320
  frame #10: libsystem_dispatch.so`_dispatch_unote_unregister_direct + 228
  frame #11: libsystem_dispatch.so`_dispatch_source_invoke + 1052

Thread #2, the main thread, was already inside exit() → __cxa_finalize. The test itself had completed. The fault is asynchronous on the manager thread, so where the main thread happens to be when it lands is arbitrary — which is exactly why CI shows the segfault at a different point in the output than hardware does, and why test_libdispatch_mach was crashing in teardown too. Same path, same bug.

Two hypotheses were killed by this backtrace, both of which had looked strong on inspection: that __sc_source_cancel was releasing the private queue it was running on, and that CFRunLoopRunInMode’s internal timeout dispatch_source_t was racing its own cancel handler. Neither is on the faulting stack.

4. CFMachPort: it exists, and it will not save us

The obvious repair is to stop inverting the layering — build the run-loop source on CFMachPort the way Apple does. The provenance is better than expected and the conclusion is still negative.

Question	Answer
Did swift-corelibs-foundation ever ship CFMachPort.c?	Yes — 9 commits at CoreFoundation/RunLoop.subproj/CFMachPort.c, through the Monterey merge
Was it deleted?	The 2024-02-01 restructure renamed it to Sources/CoreFoundation/; it was removed later
Is it there at our vendored SHA 0e20e4a7?	No — HTTP 404. Same on upstream main today
Is CFMachPort.h there?	Yes — HTTP 200, and vendored here
Did NextBSD remove it?	No. It was already gone upstream when the subtree was vendored
Recoverable?	Yes — 587 lines, Apache-2.0, no licence mixing

So the header-without-implementation asymmetry in this tree is upstream’s, faithfully mirrored. Nothing was dropped locally.

5. The CoreFoundation constraint stack

Even setting the dispatch dependency aside, the CFMachPort route is blocked three deep:

SCNotify → CFMachPort route
  ├─ CFMachPort.c absent from the vendored subtree (only 3 headers)
  ├─ CFRunLoop compiles TARGET_OS_BSD, not Mach
  │     CFTargetConditionals.h → TARGET_OS_MAC = TARGET_OS_DARWIN = 0
  │     __CFPortSet is { int kq; }; "ports" are packed pipe fd pairs
  └─ #141 §2: version-1 (port-backed) sources are NEVER SERVICED on BSD
        CFRunLoop.c:3259 has no TARGET_OS_BSD arm
        ── and CFMachPort creates a version-1 source ──

So even with the file recovered and vendored, the source it creates would never fire. #141 would have to land first, and #148 notes that its own timer rework also fixes #141’s busy-spin — the two are entangled.

One clarification worth recording, because it is easy to conflate: SCNotify contains zero xpc_ references. There is no libxpc to remove from it. The dependency under discussion is libdispatch alone, which puts this in the #153 track and not #152.

6. Why CI reported this green for days

The crash is not new. Five consecutive green runs were re-examined and every one of them segfaulted:

33469759875  fix/boot-test-fast-fail        SEGFAULT (marker absent, run still GREEN)
33468499755  fix/libnotify-drop-xpc-calls   SEGFAULT (marker absent, run still GREEN)
33468041443  fix/syslog-drop-xpc            SEGFAULT (marker absent, run still GREEN)
33468022815  fix/syslog-drop-xpc            SEGFAULT (marker absent, run still GREEN)
33467636213  fix/syslog-drop-xpc            SEGFAULT (marker absent, run still GREEN)

boot_soft: true made continue-on-error mask the step conclusion, so a missing marker never failed the job. Flipping it to false converted a long-standing silent failure into a visible one. That is the entire reason this is being looked at now.

It also produced a false attribution worth recording. The libnotify XPC-removal PR was reverted on the belief that it caused this. It did not: the crash appears on fix/libnotify-drop-xpc-calls itself, on branches predating it, and on a comment-only PR with the revert already applied. Every changed line in that PR’s SCNotify.c hunk is inside a comment block. Landing changes onto a red baseline is what made an innocent PR look guilty — while main is red, no PR’s CI result is interpretable.

7. The four surveys, answered

Question	Answer
When did it last work?	81 of 81 runs passed until 2026-08-30T02:02Z. Cause identified — §8
Where is the defect?	libmach/dispatch_kevent.c. Not the kernel, not libdispatch — §9
What about GNUstep?	A non-problem. They never meet — §10
Should we adopt CFMachPort?	No. Its core semantics are unimplementable here — §10

8. When it last worked, and what broke it

All 200 build.yml runs from 2026-07-03 to 2026-09-01 had their boot logs downloaded and read. No artifacts had expired. SC-RUNLOOP-FAIL was never emitted in any run, ever — the segfault is this test’s only failure mode.

Window	SC-RUNLOOP-OK	Segfault
2026-07-03T18:40Z → 2026-08-30T00:05Z	81	0
2026-08-30T02:02Z → 2026-09-01T04:57Z	3	17

A category the brief did not anticipate: 100 of the 200 runs never ran scrltest at all, because the suite aborted at an earlier marker. Those are inconclusive, not passes. Boot evidence also comes only from PR-branch runs — ci-image-boot is skipped on main pushes.

The transition is branch-correlated, not statistical

The first segfault precedes the last pass by 1h47m, which reads as a race until the trees are checked:

Run · SHA	Tree has the real pset release?	Result
33289506874 · 1a60ae21b	no	OK ×2
33287019802 · 0a2eb9665	yes	SEGV ×2
33291148223 · a8f47292e — F2 disabled	no, reverted to the no-op	OK
33292666690 · 763f3b998 — un-member dropped	yes	SEGV

All three post-boundary passes are exactly the trees lacking the real release; all seventeen failures are trees that have it. A clean discriminator, not a coin flip.

The change: userland #106, reg_release_pset

static void
reg_release_pset(mach_port_name_t pset)
{
	(void)mach_port_mod_refs(mach_task_self(), pset,
	    MACH_PORT_RIGHT_PORT_SET, -1);   /* was: mach_port_deallocate() — a no-op */
}

mach_port_deallocate returns KERN_INVALID_RIGHT on a port set, so nothing was ever actually destroyed before #106. The change made teardown happen for the first time. reg_release_pset on current origin/main is unchanged, so the trigger is still shipping.

Ranked suspects, and a correction

Change	Verdict
userland #106 — real port-set release	HIGH — verified from boot logs with the kernel held constant; the project’s own bisect branches isolate this half
kernel #147 — machport_filtops.f_isfd = 0	MEDIUM, unverified — knote_fdclose() can no longer reap these knotes and port names are fds. Postdates the boundary, so not the original cause, but a candidate second path (kernel #169)
userland #136 — EV_CLEAR removal	EXONERATED as introducer — the first segfault predates it by 44 hours and both Aug-30 failing runs still carry EV_CLEAR
kernel #168 — filt_machport locking	LOW — postdates the boundary by 25h. A fix landed after the fact, not a cause

9. The mechanism, end to end

Independently of the bisect, reading the source produces the full chain. Every step below is verified in code.

1. The knote carries a libmach pointer. dispatch_kevent.c:434,450 registers the native knote as EV_SET(out, r->wrap_pset, EVFILT_MACHPORT_NATIVE, EV_ADD, 0, 0, r) — udata is a struct mach_kev_reg *, never a libdispatch unote.
2. The registration is freed while the knote lives. :402-412 submits EV_DELETE and then calls reg_destroy(r) immediately — before __sys_kevent() runs at :947.
3. Releasing the pset queues an event on it. ipc_pset.c:490 does KNOTE_LOCKED(&pset->ips_note, EV_EOF); filt_machport returns 1, so the knote is activated. :491 knlist_clear() sets EV_EOF | EV_ONESHOT and leaves the knote registered, still holding udata == r.
4. The kernel copies it verbatim. kern_event.c:2220-2232 — for EV_ONESHOT, *kevp = kn->kn_kevent;. f_event is not called, so the kernel never touches the dead pset.
5. The wrapper forwards it unfiltered. dispatch_kevent.c:1039-1057 — the lookup returns NULL because r was unlinked, and the r == NULL fallback does not test the filter. The raw native event is copied out with its freed udata.
6. libdispatch dereferences it. event_kevent.c:389-394 is a bare cast, (dispatch_unote_class_t)ke->udata, with no validation. :483 loads du_type at offset 0; :490 does an atomic RMW through du_owner_wref at offset 8. On a freed mach_kev_reg those are a list pointer and an fd. SIGSEGV.

This also explains thread #3. kern_event.c:1863 returns ENOENT for an EV_DELETE with no matching knote — which is what happens once the manager thread has already consumed the EV_ONESHOT event and dropped the knote. The two threads are two halves of one sequence, not two bugs. Thread #3 survives only because _dispatch_kevent_print_error refuses to dereference udata when EV_DELETE is set.

The fix

A libmach-private filter number and a libmach-private udata must never cross the kevent_qos() boundary. The crash fix is an explicit filter test in place of the r == NULL fallback — on the order of three lines. A second, larger piece (~15-25 lines) removes the EV_DELETE on an already-freed port-set name, defers reg_destroy until after the syscall, and drops a sentinel that manufactures ENOENT on ident (uintptr_t)-1.

Under 50 lines, in one file, with no kernel change and no libdispatch change — which preserves the property that this tree builds unmodified Apple libdispatch.

The faulting kevent, decoding struct kevent_qos_s (ident 0, filter 8, flags 10, qos 12, udata 16):

0x89692550: 0x0000000000000017   ident  = 23
            0x000000008010fff0   filter = 0xfff0 = -16   ← EVFILT_MACHPORT_NATIVE
                                 flags  = 0x8010         ← EV_EOF | EV_ONESHOT
0x89692560: 0x000063efa4e01000   udata  = 0x63efa4e01000 ← identical to x0

Filter −16 is the raw-native leak; −22 would have meant the sibling bug. EV_EOF | EV_ONESHOT is the exact signature knlist_clear() stamps on the knote in ipc_pset_destroy(). And the object libdispatch dereferenced:

0x63efa4e01000: 0x0000000000000000   "du_type"       → NULL   (freed list pointer)
                0x0000000000000007   "du_owner_wref" → 7      (an fd)
0x63efa4e01010: 0x0000000000000016   → 22
                0x0000000000000017   → 23

Two port names in a struct libdispatch believes is a dispatch unote. The cross-check that closes it: 0x16 is exactly the ident from that run’s MACH_DEBUG_WRAP trace ([WRAP] change kq=9 ident=0x16), and 0x17 is this kevent’s own ident — tying the freed allocation to the very registration traced being created and deleted. The fault itself:

<+16>: ldp  x10, x8, [x0]     ; x10 = word0 = 0,  x8 = word1 = 7
<+24>: mvn  x9, x8            ; _dispatch_wref2ptr(du_owner_wref)
<+28>: ldrb w8, [x10, #0x9]   ; ← FAULTS: byte at 0x0 + 9

A NULL+9 read — dux_type(du)->dst_action, du_type at offset 0. Fix A is the correct fix; the −22 alternative is ruled out.

What the kernel change did not do

The same measurement re-ran the crash rate across the kernel boundary:

Kernel	Crashes / 30
20260831-232019 — before #167 / #168	13
20260901-032651 — with #167 / #168	9

Statistically indistinguishable (z ≈ 1.07, p ≈ 0.28). #168 changed nothing measurable, confirming from the other direction what the bisect concluded from the dates. Note also that the crashing runs print SC-RUNLOOP-OK before dying — the test passes, then the manager thread faults at exit.

10. The GNUstep question dissolves

The premise behind “bring CFMachPort in side by side with GNUstep” is that the two would collide. They never meet.

• libs-corebase — GNUstep’s CoreFoundation — is not built or shipped by anyone here. Adding it was proposed and closed unmerged.
• gnustep-base contains no CoreFoundation and links none. Its run loop is poll(2)/select(2); there is not one kqueue or kevent reference in the tree. No Mach, no CF, no CFRunLoopSource.
• No Gershwin process loads libCoreFoundation.so.6. The one /usr/lib/system library it deliberately uses, libdns_sd, has no LDADD at all.
• The genuine shared-library conflict was always libdispatch, not CF, and it is already solved by renaming this tree’s copies to libsystem_dispatch.so / libsystem_blocks.so so GNUstep’s stock names stay free.

There is exactly one real coupling, and it runs the opposite way from the proposal. Foundation.h ends with an __has_include(<CoreFoundation/CoreFoundation.h>), and this tree installs precisely that path — so every Gershwin .m that imports Foundation already pulls in these headers. It is harmless today because CFMachPort.h’s body is behind TARGET_OS_MAC and therefore empty. Enabling CFMachPort is what would inject Mach into every GNUstep translation unit on the box.

And CFMachPort could not work here anyway

Beyond the constraint stack in §5, one blocker is decisive: mach_traps.c:542-591 makes mach_port_request_notification a boot-safe no-op returning KERN_SUCCESS, because the kernel registers no such trap — returning an error makes launchd abort and panics PID 1. CFMachPort’s entire invalidation contract rides on dead-name notifications. A CFMachPort on this kernel would silently never invalidate. That is an API whose documented semantics cannot be implemented, not a missing optimisation.

Flipping TARGET_OS_MAC is likewise not a config change but a source fork: it pulls in Darwin malloc zones (CFBase.c redefines CFAllocator to match struct _malloc_zone_t), Mach-O headers, crt_externs, vouchers, mk_timer_* traps, and mach_port_insert_member/extract_member — which libmach does not implement. It also registers __CFMachPortClass, a symbol with no definition in the tree: link failure on day one. And it silently breaks the version-1 perform calling convention, because the library gates arity on TARGET_OS_MAC while the public header gates it on TARGET_OS_OSX || TARGET_OS_IPHONE.

11. What follows

1. Fix the wrapper — under 50 lines in dispatch_kevent.c. It fixes the crash for every Mach recv source in the system, not just scrltest.
2. Confirm cheaply first — revert reg_release_pset() to mach_port_deallocate() and rerun. If the marker returns, #106 is proven; if not, kernel #147/#169 is a live second cause.
3. Fix #141 separately — adding || TARGET_OS_BSD to the #elif at CFRunLoop.c:3258. The registration side is already live on BSD and not Mach-gated; only the dispatch arm omits the platform. No type change, no header change, no SOVERSION bump, and upstreamable.
4. Do not port CFMachPort, and drop “side by side with GNUstep” as a constraint.
5. Two upstream bugs worth filing at swift-corelibs — the shadowed timeout at CFRunLoop.c:690 that makes every BSD run loop busy-spin, confirmed present upstream and not a local regression; and the 16-mode cap that crashes with “Unable to create timer Port”.
6. Gate CFMessagePort.h and CFUserNotification.h — both installed and in the umbrella header with no .c behind them: a clean compile followed by an undefined-symbol link failure.

A process note: #158, which tracked this crash, was closed 2026-09-01T04:46Z with no fix landed — and run 33471785512, eleven minutes later, still segfaults on both arches.

Evidence: core dump and 30-run stress on arm64 hardware running the CI main userland (603 files, dc635ddf); Apple sources from apple-oss-distributions/configd tags configd-212.2 and configd-289; CFMachPort.c recovered from swiftlang/swift-corelibs-foundation at d8e8a8b92b. The hardware kernel was 20260831-232019 and could not be brought to CI parity — the nextbsd-kernel-arm64 CI artifact is a build-object tree with no linked kernel and no modules, which is its own defect.