← Back · companion to MIG restoration, EVFILT_MACHPORT, launchd → Mach
Mach messages are lost non-deterministically on NextBSD. A client’s send succeeds, the server never receives it, and the client blocks forever. This page records what was measured on hardware, what was ruled out, and what is left.
Priority. This is a defect in the IPC substrate itself, not in any one daemon. Every Mach service inherits it. It should be fixed before further userland porting, because it makes every other component’s behaviour unreproducible.
Read §1 and §1b with this caveat. Every measurement on this page below was taken with the box running test/78-notifyd-plus-127 — an experimental userland arm that was measured at 25–30% and explicitly recommended against merging. It was deployed during an experiment and never reset, so it was in place for all of them.
The true baseline, both repos at main: 11 failures / 300 sends (a second run gave 16/300) — roughly 4–5%, scattered and persistent. Cold boot, kernel 20260831-040641.
| old userland (#127+#129) | main userland | |
|---|---|---|
| shape | burst, then clean | scattered, uniform |
| block 1 | 4–8 failures | 0–2 failures |
| converges? | yes, hard, at ~35 | no — still failing at block 30 |
| total / 300 | 35 | 11–16 |
The traffic-decay signature, the warm-up, and the fixed budget all describe that userland, not this system. On main there is no decay to explain: failures arrive at a low, roughly constant rate and never stop. A constant rate reads as a race, not as a resource being exhausted — so the mechanism narrative built around decay does not carry over.
The underlying defects documented below (no generation counter, names are file descriptors) are real and independently verifiable. They are simply no longer supported by the measurement that motivated them. Anyone continuing this work should re-derive the mechanism against the ~4% persistent failure on main rather than inheriting the reasoning here.
A client calls mach_msg(SEND|RCV). ktrace of a wedged client shows the trap entered and never returning:
syslog CALL lkmnosys(0x332f99d21368,0x3,0x24,0x40,0x12,0)
syslog RET lkmnosys 268451845/0x10004005 <- MACH_RCV_INTERRUPTED, from our SIGKILLArgs decode as option=0x3 (SEND|RCV), send_size=36, rcv_size=64, rcv_name=0x12. The send half succeeds; the client then sits in ipc_mqueue_receive indefinitely.
| Property | Established by |
|---|---|
| Genuine permanent wedge, not slowness | 90-second timeout changes nothing; 5 of 13 sends burned the full 90s |
| Bimodal — sub-second success or never | timed probes; no intermediate durations observed |
| Server never sees the message | marker never appears in the log; search method validated against a successful send |
| Client and server both blocked simultaneously | procstat -kk: client in ipc_mqueue_receive, server thread also parked in ipc_mqueue_receive |
| Traffic-driven decay | fresh boot 40–60% wedge; after ~70–80 cumulative sends → 0, and stays 0 |
| Time does not help | 10 minutes idle, zero sends → still 15/30 wedged |
| Resets when either daemon restarts | warm (0/20) → restart syslogd → 4/20; re-warm → restart notifyd → 10/20 |
The decay-then-stick pattern is the most diagnostic property: whatever is wrong is exhausted or outgrown by allocation, and is rebuilt when a daemon’s port table is rebuilt.
Exactly 35 failures. Every time. Then zero, forever.
Measured across four different kernels, each with a completely different distribution:
arm per-block total
baseline [4, 6, 3, 7, 6, 6, 3, 0, 0, 0] 35
+#144 [7, 9, 8, 5, 6, 0, 0, 0, 0, 0] 35
+Stage 1 [4, 3, 4, 5, 6, 10, 3, 0, 0, 0] 35
+alloc_name [8, 7, 10, 10, 0, 0, 0, 0, 0, 0] 35Then confirmed directly, 25 blocks of 10 from a cold boot:
block 1..6 3, 8, 8, 6, 5, 5 cumulative 35
block 7..25 0 every block cumulative still 35
TOTAL: 35 failures out of 250 sendsThe counter reaches 35 and does not move for a further 190 sends. This is not a probability — it is a fixed budget that gets consumed and is then gone.
Every arm previously reported as “35/100, unchanged” was read as this change did nothing. The stronger reading is that something with a capacity of ~35 is consumed, and none of these changes touches it — they only redistribute when the failures land.
It also means the percentage framing used in the earlier arms was misleading. “35/100 = 35%” invites comparison against 0%; the honest statement is “35 failures, then clean”, and the denominator was a choice of how many sends to make, not a property of the system.
The shape fits everything else: traffic-driven rather than time-driven (each send consumes budget), unaffected by idling, and reset by a daemon restart — because the restart re-creates whatever the budget is made of.
Each of these was a live hypothesis, tested and eliminated. They are recorded because knowing where the bug is not is most of the value.
| Hypothesis | Verdict | Evidence |
|---|---|---|
| notifyd crashing mid-RPC | Refuted | 0 aborts during a burst where 10/12 sends failed; PID stable. The 18 logged aborts were historical — the log spans two days and is never truncated. |
libnotify shm_base NULL causing RPC fallback | Refuted | that path returns NOTIFY_STATUS_SHM_BASE_NULL; it cannot hang |
notifyd has no shm slots (nslots==0) | Refuted | segment exists, 4096 bytes, 1024 slots (current id 39) from notifyd’s own status dump |
| EVFILT_MACHPORT bridge missing wakeups | Refuted | notifyd-only RPCs: 1 wedge in 55. During a wedge notifyd is fully idle and healthy. The wedged path used a raw blocking mach_msg_receive, not the kqueue bridge. |
| launchd does not hold the MachService receive right upfront | Refuted | machservice_setup() (called from plist parsing) → machservice_new() → launchd_mport_create_recv(), ms->recv = true, ms->upfront = true. Apple’s semantics are implemented. |
| It is really just slowness, manufactured by short timeouts | Refuted | 90s timeout — 15× the original — changes nothing |
| libxpc is the root cause | Not on the failing path | asl.c, the client send path, has zero XPC call sites; traces show pure MIG/Mach |
| Mach syscall slot wiring | No evidence | slots are dynamically allocated and resolved via sysctlbyname("mach.syscall.<name>"); ktrace printing lkmnosys is cosmetic (unnamed slot) |
Status: strongly supported by reading, NOT confirmed by measurement. The generation counter was implemented and merged, and the resulting kernel did not boot (see the plan’s progress section) — so it has never actually run. Until it does, this section is the best available explanation, not a demonstrated cause. The fixed-budget result in §1b is a fact; this is the hypothesis that fits it.
Confirmed by reading, in src-overlay/sys/sys/mach/port.h:
#else /* NO_PORT_GEN */
#define MACH_PORT_INDEX(name) (name)
#define MACH_PORT_GEN(name) (0)
#define MACH_PORT_MAKE(index, gen) (index)
#endif /* NO_PORT_GEN */Classic Mach encodes a generation count in every port name precisely so that a reused name is detectably stale. Here it is compiled out. IE_BITS_GEN survives in ipc_entry.h but appears in exactly one .c file — inside a debug printf.
Names are raw file descriptors from kern_fdalloc(td, 16, &fd) (ipc_entry.c:453) — lowest-free-at-or-above-16 — and every free resets the search downward (ipc_entry.c:893-894). A name released at the end of one RPC is handed back bit-identical on the next, and nothing in the kernel or userland can tell that a cached name now denotes a different object.
Two confirmed leaks march the fd low-water mark monotonically upward:
ipc_entry_alloc_name() allocates the fd twice (ipc_entry.c:559-582). It reserves fd == name, installs a falloc_noinstall placeholder with badfileops there, then calls ipc_entry_get(), which ignores the requested name and allocates a different fd. The placeholder is never dropped, so that name is burned for the life of the process — and the returned entry’s real name differs from the one later stamped into port->ip_receiver_name.fp->f_count with an inconsistent +1 (ipc_entry.c:618-622). A one-uref send right has f_count == 2, so the final-release branch in ipc_right_dealloc (ipc_right.c:809) is unreachable: mach_port_deallocate on a send right never frees the name.Early in a daemon’s life the recycled band and the live band overlap, so collisions are common. As the leaks push the low-water mark up, the bands separate and collisions stop — permanently, because fds never go back down. Restarting a daemon resets the table to 16 and recreates the dense band. Elapsed time is irrelevant because the driver is cumulative allocation, not time.
That accounts for every measured property in §1: traffic-driven, time-independent, bimodal, resets on daemon restart.
ipc_object_copyin() wraps any non-Mach fd into a synthetic port rather than returning an error (ipc_object.c:411-412, ipc_entry.c:355-386): it allocates a special port, sets ip_receiver, and returns success. A recycled name that now denotes a vnode or socket silently becomes a port with nobody receiving on it. The send succeeds, the server never sees the message, and the client blocks forever awaiting a reply — which is precisely the observed state.
ipc_mqueue_deliver() consulted only one of the two thread pools:
receiver = NULL;
/* first we check the the port and portset for waiters */
if (pset != NULL) {
receiver = thread_pool_get_act((ipc_object_t)pset, 0);
} else if (receiver == NULL) { /* dead branch */
receiver = thread_pool_get_act((ipc_object_t)port, 0);
}receiver was set NULL immediately above, so the second arm was just else: a port belonging to a set never had its own pool checked, and the #369 re-check block below repeated the omission. The fallback ipc_pset_signal() only activates EVFILT_MACHPORT knotes and, per its own comment, “since #256 it never wakes pool receivers.”
Our userland creates exactly the triggering shape: libmach/dispatch_kevent.c:272 allocates a port set and :279 moves service ports into it, while syslogd’s database_server thread receives on the bare port name. Fixed in nextbsd-kernel PR #144. This is unconditional and real, but it does not explain the traffic decay — the generation counter does.
Found by a four-way audit of libnotify/notifyd; each verified independently against the code or the running machine. None of these is the wedge, but all are real.
| Defect | Consequence |
|---|---|
os_atomic_inc/dec map to atomic_fetch_add/sub, returning the pre-op value where Apple’s return post-op (freebsd-shims/os/atomic_private.h:13-14) | Every refcount release in libnotify is off by one. notify_cancel() never actually cancels; nodes are never freed; a genuine over-release now passes the assert meant to catch it and frees a live node. |
SYS_initgroups defined as 81 — which is getpgrp on this system (verified against /usr/include/sys/syscall.h: getgroups=595, setgroups=596) | notifyd silently never restores its supplementary groups; drifts to the last client’s credentials. |
pthread_setugid_np is a no-op returning success | notifyd’s only path access-control check is defeated; stat() runs as root, so any client can watch any file. |
sandbox_check_by_audit_token returns 0 (allowed) unconditionally | Fails open on three security decisions, including one used as a sandbox detector. |
O_SYMLINK mapped to O_NOFOLLOW | Opposite operations — symlink watches can never work. |
NOTIFY_INTERNAL_CRASH is a returning no-op | Four kern_return_t MIG routines fall off the end and return garbage to the demux. |
Every registration on the running system is plain; memory 0 (notifyd status dump) | No registration is shared-memory backed, so every notify_check is an RPC. This is a consequence of RPCs not completing, not a cause. |
Recorded so the same ground is not re-covered.
reached_server column grepped a never-truncated log by the current syslogd PID — and syslogd gets the same PID every boot, so the count accumulated across the whole run and read as a rising per-send tally. It is a per-boot 0/1 flag. Fixed: the log is now truncated before each reboot. The ok/attempted totals were never affected.vm.uma.ipc_kmsg_zone reads zero during a wedge — but its allocs counter also stays zero through real traffic, so this kernel does not route messages through that zone. The reading proved nothing.ipc_entry.h; it needs stamping in ipc_entry_get() and checking in ipc_entry_lookup(). Until a reused name is detectably stale, Mach IPC on this system is not deterministic.fdalloc in ipc_entry_alloc_name() and the f_count/uref off-by-one. Note these currently mask the collision window by pushing the fd band upward — fixing them without (1) will make the failure more frequent, not less. Sequence matters.ipc_object_copyin() reject non-Mach fds instead of wrapping them in a receiverless port.os_atomic_inc/dec.Ordering warning. Item 3 must not land before item 1. The leaks are what currently drive the failure rate to zero after ~70–80 calls; removing them while names are still ungenerationed would keep the recycling band dense indefinitely and make the wedge permanent rather than transient.
Both were considered. Neither is supported by the evidence as a fix for this bug:
asl.c: zero XPC call sites). Removing it is a legitimate debt-reduction goal — our libxpc is 7,143 LoC of FreeBSD nvlist-based reimplementation, not Apple’s — but it would not fix this.lkmnosys band (slots 599–646). The XNU-style separate trap table is already scoped and deferred there.