← Back · supersedes the mechanism sections of Mach IPC reliability · see also namespace separation
Six mechanisms measured clean, seven hypotheses dead, and a bug that is still open. This is the negative-results page — it exists so the next person does not re-tread ground that has already been settled by measurement.
A synchronous Mach round trip hangs, permanently, at a low rate.
| property | evidence |
|---|---|
| Rate | ~4–5%. Of 120 sends: 110 ok, 10 hung, 0 other failures. |
| Genuine hangs | every failure is rc=124 at exactly 6s; an earlier run survived a 90-second timeout unchanged |
| Flat, never converges | 30 blocks × 10: 0 1 0 2 1 2 0 0 1 2 0 1 1 0 0 0 2 0 0 0 1 1 0 0 1 0 0 0 1 0 |
| Rate-sensitive | back-to-back 4/60 vs 1s apart 1/60 |
| Concurrency does not hurt | 4 parallel workers, 60 sends, 0 failures |
| Vanishes under tracing | ktrace and LAUNCHD_TRACE runs both stopped reproducing it |
| Not notifyd | with notifyd 0/40, without notifyd 0/40 |
Each syslog -s performs ~27–32 Mach RPCs. Solving 1-(1-p)^27 = 0.04 gives p ≈ 0.0015: roughly one RPC in 700 hangs, and it can be any of them. This was never a logging bug — which is why every path-specific fix measured exactly nothing.
These are not arguments. Each is a counter compiled into the kernel and read against a workload that reproduced the failure.
| mechanism | counter | result |
|---|---|---|
| Lost wakeup, port-set receive | mach.lost_wakeups | 0 (four runs) |
| Lost wakeup, bare-port receive | mach.lost_wakeups_bare | 0 |
| Silent drop, inactive destination | mach.send_dead_port | 0 |
| Silent drop, circular message | mach.send_circular | 0 |
| Handoff wakeup lost | handoff_posted vs handoff_received | no positive gap |
| Delivery accounting | deliver_calls / _handoff / _enqueue | 1050+309 vs 1357 — reconciles |
The last one matters as a control: it shows the counters themselves are trustworthy, because the two arms add up to the call count.
mach.lost_wakeups instruments only ipc_mqueue_pset_receive(). The client waits for its reply on a bare port, so three clean runs of it said nothing about where the hang manifests. That is why lost_wakeups_bare exists. Both are blind by construction to the direct handoff path, where ipc_mqueue_run() takes the message off the queue and into the receiver’s ith_kmsg — which is why the posted/received balance exists. Each counter was added to close a blind spot in the previous one.
| # | hypothesis | killed by |
|---|---|---|
| 1 | notifyd is causal | with/without both 0/40 on main. It was causal on the old userland — that result did not generalise. |
| 2 | EVFILT_MACHPORT bridge misses wakeups | notifyd-only RPCs: 1 hang in 55; notifyd idle during a wedge |
| 3 | launchd lacks upfront MachService receive rights | source: machservice_setup() creates them at plist-parse time |
| 4 | Port names recycled without generations (decay model) | the decay signature was an artifact of a bad userland; on main nothing converges |
| 5 | Bare-port receiver on a set member never woken (#144) | database_server receives on the set, so the precondition never holds |
| 6 | Send silently dropped to a dead port | send_dead_port = 0 over 120 sends |
| 7 | Handoff wakeup lost | posted/received balance shows no gap |
Section 4 previously said misrouting of the reply was the last candidate. That is now disproved. The reply path was never the problem: the request never gets processed, so no reply is ever generated. Every instrument up to that point had been aimed at the client’s reply port — the wrong end of the RPC, which is exactly why they all read clean.
A client caught mid-hang is parked in the kernel, not in userland:
392 I thread_b syslog -s -l Notice VALIDATE-PROBE
thread_block+0x44 ipc_mqueue_receive+0x3c0 mach_msg_receive+0xd8 sys_mach_msg_trap+0x40It sat there for 226 seconds and completed the instant one unrelated request was sent to the same service. The request had been sitting on the server’s port the whole time; the later message re-triggered the wakeup and drained both.
mach.port_backlog walks every space and reports each port holding an undelivered message, with the process owning the receive right. During a live wedge:
pid comm name port msgcount pset waiters rights recv-right-owner
131 syslog 20 ffffa000c40bf680 1 yes no S notifyd[31]
32 syslogd 22 ffffa000c40bf680 1 yes no S notifyd[31]
31 notifyd 19 ffffa000c40bf680 1 yes no R notifyd[31]
1 launchd 28 ffffa000c40bf680 1 yes no S notifyd[31]One port, one message, four spaces. notifyd holds the receive right — and notifyd has no thread in ipc_mqueue_receive at all. Its Mach traffic is serviced entirely through EVFILT_MACHPORT on a kqueue.
| Observation | Explanation |
|---|---|
| Vanishes under tracing | Tracing perturbs timing and adds traffic that re-triggers the wakeup |
| Zero failures under concurrency | Concurrent sends continuously nudge each other |
| Rate-sensitive (4/60 vs 1/60) | The slower the traffic, the longer a stranded request sits |
| Flat ~4%, never converging | A race on an idle path, not a resource that degrades |
An isolated send on an idle system is precisely the one that strands, because nothing follows it to knock the server loose.
aslmanager’s service port holds an undelivered message from boot, on every boot, across every kernel measured. Same signature: kqueue-serviced, no Mach receive thread. It requires no race to observe, which makes it the best starting point for the remaining work.
The cause was one flag in userland. dispatch_kevent.c registered the EVFILT_MACHPORT readiness knote EV_ADD | EV_CLEAR, while the Phase 3 ERROR_EVENTS skip assumed level-triggered semantics — its own comment says “the next ordinary poll delivers it”. With EV_CLEAR the kernel consumes the readiness when it hands the event over (kern_event.c:2270), and nothing re-arms it, so the message waited on the port for unrelated traffic. Dropping EV_CLEAR makes the assumption true (PR #136).
| test | before | after |
|---|---|---|
| original 8-boot harness | 22/80 | 80/80 |
| aslmanager frozen at boot | every kernel measured | gone |
syslog -s failures | 9, 12, 5, 5 per 150 | 0, 0, 0 per 150 |
| indefinite strand, 60 attempts | caught at 2–31 | none |
The raw harness reads 72/80, and that understates it. Every boot scored exactly 9/10 — not the shape of a random 4–5% fault. Per-send timing showed the first send after boot takes ~13.5s and succeeds, with the message confirmed in system.log; the harness’s 10s timeout was simply shorter than a slow cold start. That residual is a separate deterministic latency bug, filed as userland #137. Delivery is 80/80.
A criterion set in advance failed, and it was the criterion that was wrong. The commit required the kernel-side notify/drain gap to collapse toward 0; it rose from 10/12/11/6 to 26/24/29. Level-triggering re-reports the same readiness until drained, so rcvlarge_notify now counts re-notifications — psetport_drained is flat (~897 → ~907) while notify rose ~27. Nothing is lost; the metric stopped measuring loss. The deterministic aslmanager test and the failure counts are what carry the result, and both were fixed in advance.
Two counters settle where the loss happens. rcvlarge_notify counts “a message is waiting on port N” events handed to userland; those do not dequeue anything, since filt_machport() returns with the message still queued and leaves userland to issue its own mach_msg(). psetport_drained counts messages actually taken off a port-set member port.
| run | notifications | read | gap | client failures |
|---|---|---|---|---|
| 1 | 919 | 909 | 10 | 9 |
| 2 | 901 | 889 | 12 | 12 |
| 3 | 906 | 895 | 11 | 5 |
| 4 | 901 | 895 | 6 | 5 |
The gap is always positive: userland is told about ~1% more messages than it reads. Because the knote is EV_CLEAR, nothing re-arms it, so an unread message waits for unrelated traffic to signal the set — which is exactly the nudge behaviour that opened this investigation.
The gap is the same order as the failure count but not a strict 1:1 — run 3 is 11 against 5. An undrained notification only becomes an observed hang when a client is waiting on that particular reply. The defensible claim is the weaker one.
Carried to userland #135. The kernel side is finished.
Recorded because each of these was stated with more confidence than it deserved.
test/78-notifyd-plus-127, an experimental arm measured at 25–30% and explicitly recommended against merging. The absolute figures from those arms are not a baseline.MACH_SEND_INVALID_DEST in launchd is benign. Filed as a fork/Mach-state bug, then closed: PID 1 nulls its own bootstrap_port deliberately, in vendored Apple code, and the forked child inherits the NULL.lost_wakeups_bare answers a narrower question than its name. It samples ip_msgcount before parking, under the same lock as the queue check above it, so it can only see a message that was already there. A message enqueued after a thread parks is invisible to it, and its zero readings never excluded that case.kqueue_register() does call f_event after f_attach, which was the check the ticket itself named. #166 (a missing in-flux guard) measured 0 occurrences and is recorded as latent rather than as the cause.EV_CLEAR clears KN_ACTIVE|KN_QUEUED after a successful report exactly as after a zero return, so status=0x0 is consistent with both “told” and “not told”. It did reveal EV_CLEAR, which is what made the notify-vs-drain count the right next measurement.fix/pset-scan-stale-msgcount and fix/pset-signal-kqwakeup. Both looked correct from code reading; both had counters that came back zero. One of them passed its strand test purely by luck while provably never executing.Each measured on a run that reproduced the failure:
| Mechanism | Reading |
|---|---|
| Send-path blocking | 0 entries in 150 sends |
| Threads stuck parked | enter/exit balanced |
| Scan missing a queued message | 0 finds across 3118 second passes |
| Wakeup not delivered | 897 sleepers found, 897 woken, of 901 enqueues |
| Port-set identity (recycled name) | 0 mismatches in 1823 evaluations |
| Name translation failure | 0 |
| Notification’s port name wrong or null | 0 nulls in 920 notifications; ip_rcvname matches the entry name |
| Dead-port and circular drops | 0 |
Every instrument that encoded an assumption went silent and taught nothing. Every instrument that just counted was right.
ith_state and object type — produced nothing across two confirmed hangs.Against that: six counters, six unambiguous answers, all of which remain true regardless of what the bug turns out to be.
Two pieces of tooling also proved decisive and did not exist at the start: a serial console (TCP 127.0.0.1:1234; arm64 needs console="eficom,efi", not the x86 comconsole), and hardware verification before merge — CI’s boot smoke-test passed a kernel that then failed to boot the machine.
EVFILT_MACHPORT consumer. Start from aslmanager’s permanently frozen port: no race needed, present on every boot, never resolves. Check whether the handler always issues the follow-up mach_msg() for the reported port, whether an error from it is swallowed, and whether anything re-arms after EV_CLEAR if the receive does not fully drain._asl_evaluate_send drives ~27 RPCs through this exact path per log line; cutting them shrinks the exposure proportionally before any real fix.mach.port_backlog. It named the culprit outright when nine counters had only excluded things.