NextBSD & Gershwin: uniquely name the base libdispatch fork

Why the menu broke (the short version)

The Gershwin global menu bar is supposed to switch to whatever app you bring to the front. On NextBSD it did not: it latched onto the first app and stayed there. The menu learns about app switches by watching the X11 display for a "which window is active now" event. It does that watch through a libdispatch file-descriptor read source — a small piece of plumbing that says "wake me when this socket has data."

The root cause was one compile-time number. NextBSD's libdispatch is a fork of the same code Gershwin uses, byte-for-byte identical except for a single flag, EV_UDATA_SPECIFIC, which the fork set to 0x0100 so that Apple's Mach messaging support would compile. The problem: on FreeBSD (which NextBSD is built on) the value 0x0100 already means something else to the kernel — "fire this once, then delete it." So the menu's watch fired exactly one time and then the kernel threw it away and never re-armed it. The menu got its first app-switch notification and then went deaf. That same flag also switched libdispatch into a Darwin-only bookkeeping model that mis-identifies file-descriptor watches on FreeBSD, compounding the failure. Timers and Mach-port plumbing were unaffected, which is why the rest of the system still booted — only file-descriptor watches like the menu's broke.

People tried several fixes. Stripping just the "fire once" bit made a single watch re-arm in tests, but did not fix the deeper wrong-bookkeeping problem. Setting the flag all the way back to zero fixed file-descriptor watches but would have broken Apple's Mach messaging (libxpc), which genuinely needs the Darwin model. The clean answer everyone converged on: keep the Darwin model ONLY for the one Mach-messaging case and route everything else through the portable path.

The famous "log out and back in fixes it" clue was a red herring. The login screen starts one X display at boot and never shuts it down on logout, and the old session's "active window" value is left behind on that still-running display. A second login inherits that stale value, so the menu latches once and looks fixed — but app-switching is still dead. A truly fresh display has no leftover value, so the menu shows nothing. It was a mask, the same way manually restarting the menu process temporarily "fixed" it.

How did we end up with TWO libdispatches? Rather than wait for the surgical base fix, the Gershwin side shipped its own libdispatch built with Mach turned OFF (which sidesteps the poisoned flag) and installed it so Gershwin apps load it ahead of the system copy, while system daemons keep the Mach-on copy. Two builds of the same library, kept apart by load order. That is the split. The base per-path fix — restricting the Darwin model to Mach-only and stripping the bad bits from everything else — was written three days later and is still sitting uncommitted in the working tree. Once it is committed and proven, the reason for the second libdispatch goes away.

The plan

Keep both builds; give the base fork its own name so the two can never collide. Gershwin is left entirely alone — it runs plain upstream libdispatch, exactly as on FreeBSD/Linux.

1. Rename the base NextBSD libdispatch to the unique soname libsystem_dispatch.so (a build-harness change — a soname flag in build-userland.sh or patchelf --set-soname). It stays a Mach (HAVE_MACH) build; only its name changes. It is a permanent fork — never synced with Apple again — so this is a clean, owned rename.
2. Relink the base -ldispatch consumers (libxpc, launchd, notifyd, SystemConfiguration, IOKit, notify, asl, plus CoreFoundation and the daemons) onto libsystem_dispatch.so.
3. Leave Gershwin untouched: it keeps building/linking the stock Apple libdispatch.so — no rename, no NextBSD special-casing, no patches. It becomes the only libdispatch.so on the system.
4. Add a CI guard: assert no Gershwin binary records a DT_NEEDED on libsystem_dispatch.so or a Mach-cluster lib — the one rule that keeps the two builds out of the same process.
5. Confirm the payoff: the two builds now have different sonames, so the launchctl/dbus/sshd “Undefined symbol” poisoning can’t happen, and Gershwin runs pure upstream libdispatch.

Note: no libdispatch source is patched on either side. Gershwin’s is stock Apple; the base fork is used as-is (Mach) and merely renamed. The earlier event.c/event_kevent.c kqueue experiments are irrelevant — they never made the menu track, and none of this depends on them.

How to read the per-library disposition

A "disposition" is simply the decision we have reached about one specific library: given how that library works and how (or whether) Gershwin uses it, which build of libdispatch should it link against, and what — if anything — do we need to change. Think of it as a one-line verdict per library so a maintainer can scan the whole system at a glance instead of re-deriving the reasoning each time. Each row answers four questions. "Needs Mach?" asks whether the library genuinely requires Apple's Mach messaging support inside libdispatch — that is true only for libraries that create a Mach-receive dispatch source (the plumbing behind libxpc and friends); everything else only needs portable timers and file-descriptor watches. "Gershwin uses" asks whether a Gershwin desktop/GUI process actually loads this library — this matters because the two-runtime hazard only bites when two different libdispatches could end up in the SAME process, so a library no Gershwin app ever loads is low-risk no matter what it needs. "Recommended dispatch" is the payoff: which single libdispatch this library should bind to — under Option A the answer is almost always "the one unified base libdispatch," because that build serves both Mach daemons and portable GUI apps from a single file. "Action" is the concrete to-do: usually "no change" for daemon-only libraries, and "delete the Gershwin-specific workaround" for the handful of pieces that currently exist only to prop up the second libdispatch.

Needs Mach?

Does this library truly need Apple’s Mach messaging inside libdispatch? Only libraries that create a Mach-receive source (the plumbing behind libxpc) do.

Gershwin uses

Does a Gershwin desktop app load this library? That is what makes the “two builds in one process” hazard possible.

Which libdispatch

Which build it links: the renamed base fork libsystem_dispatch.so, or Gershwin’s stock libdispatch.so (or, for libdns_sd, none at all).

Action

The concrete to-do — usually “relink onto the renamed base fork” or “no change.”

What about mDNS / the Network browser?

There is no real contradiction here; the documents are describing different layers. The audit asked: "does the mDNS (Bonjour-style network discovery) client drag libdispatch into a Gershwin app?" The answer, confirmed against the current source, is No. The client library, libdns_sd, is a plain Unix-domain-socket stub — it talks to the mDNSResponder daemon over a local socket at /var/run/mDNSResponder using ordinary sendmsg calls. It contains zero Mach code and zero dispatch code, because its dispatch paths are compiled out unless the target is Apple (_DNS_SD_LIBDISPATCH is 1 only on __APPLE__, and this build targets FreeBSD). GNUstep drives it the old-fashioned way: it takes the socket file descriptor and polls it with select() on the NSRunLoop. So dispatch never enters a Gershwin GUI process through mDNS. Only the out-of-process daemon links Mach dispatch, and a daemon in its own process cannot cause the two-runtime hazard.

The mdns-network-browse-mach-plan.html document is not a counter-example. It agrees libdns_sd is today an AF_UNIX, non-Mach, dispatch-free stub — the Mach MIG transport it describes is a PROPOSAL, not something that exists. There is no .defs (MIG interface) file anywhere in the tree, and mach_bridge.c is a bare skeleton that only registers a name and returns; it never serves a Mach message loop. The network-crash doc is a third, separate layer: a libs-base build-config bug where mDNS support was disabled (HAVE_MDNS=0), leaving a nil backend that crashed; its fix (adding -L/usr/lib/system so the link probe passes) is already present in Install-System-Domain.sh.

Bottom line for Option A: mDNS as-built is fully compatible — no Mach, no dispatch in-process. The one forward-looking flag is that IF the mach-plan's client side were ever implemented, it WOULD inject a Mach surface directly into Gershwin GUI processes, which is exactly the hazard Option A exists to prevent. So Option A should explicitly mark the Mach-MIG mDNS client proposal as "re-review before adopting," not fold it in.

Risks & how we handle them

• Missing a base consumer when relinking: if any base binary still asks for libdispatch.so, that name now belongs to Gershwin's STOCK (non-Mach) build, so the daemon loads it and dies with 'Undefined symbol _dispatch_source_type_mach_recv'.
  Mitigation: Relink ALL base -ldispatch consumers to libsystem_dispatch.so in one pass; readelf-verify every base binary records a DT_NEEDED on libsystem_dispatch.so and none on libdispatch.so.
• A Gershwin GUI process pulling in a base library that now needs libsystem_dispatch.so would load two dispatch builds in one address space — the exact hazard.
  Mitigation: The only base lib Gershwin consumes, libdns_sd, is dispatch-free. Add a CI check that no Gershwin binary records a DT_NEEDED on libsystem_dispatch.so or on a Mach-cluster lib (libxpc, launchd, notifyd, SystemConfiguration, IOKit, notify, asl).
• BlocksRuntime must stay a single shared copy: two copies would split the _NSConcreteStackBlock / _NSConcreteGlobalBlock class symbols and break blocks passed between the two dispatchers.
  Mitigation: Keep one libBlocksRuntime.so system-wide; do not fork it alongside dispatch.
• The Mach-MIG mDNS client proposal, if ever built, would inject a Mach surface straight into Gershwin GUI processes.
  Mitigation: Keep libdns_sd a dispatch-free AF_UNIX stub; CI-check it has no dispatch/xpc DT_NEEDED. Re-review the Mach-MIG client plan before adopting, don't fold it in.

Open questions

• Confirm the full set of base -ldispatch consumers to relink against libsystem_dispatch.so (the ~19 Makefiles: libxpc, launchd, notifyd, SystemConfiguration, IOKit, notify, asl, CoreFoundation, the daemons, tests).
• Soname for the fork: libsystem_dispatch.so (Apple-canonical, already a symlink in the base) vs libmachdispatch.so (more self-documenting) — maintainer's call.
• Where is the rename applied — a soname flag in build-userland.sh's cmake step, or patchelf --set-soname at install? Either is fine since the fork is permanently owned; pick one and pin it.
• The six base components that require Mach sources (per 'Is the Mach fork still needed?') keep working because the fork stays HAVE_MACH — it is only renamed. Confirm nothing else in the base silently depended on the libdispatch.so NAME.