← Back · implements the fix identified in Mach IPC reliability · see also EVFILT_MACHPORT, Launchd → Mach
Stop Mach port names being file descriptors. This is the single deviation from XNU that causes non-deterministic Mach message loss on NextBSD, and undoing it is a two-stage kernel project.
Priority. Mach IPC drops messages non-deterministically. Every daemon inherits it, and it makes every other component’s behaviour unreproducible — every userland measurement taken so far was contaminated by it. This outranks further userland porting.
| Stage | Ticket | State | Measured |
|---|---|---|---|
1 prereq — ipc_pset_alloc_name init | #146 | merged | — |
| 1 — EVFILT_MACHPORT off the fd table | #147 | merged | 35/100 (unchanged, as predicted) |
2 / C0 — reverse hash off is_table | #148 | merged | — |
userland — MACH_PORT_INDEX corrected | userland#131 | merged | — |
| 2 / C1 — locking discipline | #149 | folded into C2b | — |
| generation counter (partial C2b) | #160 | merged, then REVERTED — does not boot | n/a |
| fileport generation guard | #161 | open, inert until generations return | — |
| 2 / C2a, C2b, C3–C6 | #150–#155 | open | — |
Stage 1 is done and is behaviour-neutral, which is the correct result. Kernel 20260831-021508 on arm64 measured 35/100 — identical to the pre-fix baseline and to #144. It removes the kqueue filter’s dependence on file descriptors; it does not change what a name is, so it cannot fix a stale-name bug. What matters is what did not happen: f_isfd = 0 boots, every service reports status 0, and the two changes that would have surfaced as a use-after-free or a “recursed on non-recursive mutex” panic are holding under a real launchd/libdispatch workload.
A userland change had to land first, and it was not obvious. Kernel CI boots against the published userland, so a kernel-only generation change would pass its own smoke-test and still break the box: launchd sized mig_cb_table by the raw port name, and a generationed name would ask PID 1 for hundreds of megabytes. userland#131 fixes that and is deliberately a no-op today (generation bits are zero, so MACH_PORT_INDEX(name) == name).
It also repaired a latent bug: MACH_PORT_INDEX was (name) & ~0xff, which returns zero for every port this kernel allocates — so launchd’s HASH_PORT and libdispatch’s VL_HASH put every port in bucket zero.
The generation counter passed CI and did not boot on hardware. Merged as #160, deployed as 20260831-025635; the machine never came back — no ssh, no ping, and a clean stop/start did not help. Recovered via LiveCD by restoring the Stage 1 kernel, which came back in 10 seconds with every service healthy. Reverted in #162 because main otherwise ships an unbootable kernel.
Two process failures, each worth more than the patch:
utmctl attach is not implemented — it prints a warning and echoes the controlling terminal. No panic was recoverable, so a panic cannot be distinguished from a hang, and every failed boot costs a full LiveCD recovery cycle that teaches us nothing. Getting a console is now a prerequisite for C2b, not a nicety.Prime suspect, unverified: ipc_entry_alloc_name() is reachable from ipc_object_copyout_name() on the live mach_msg path and already double-allocates, returning a name that does not match the one requested. #160 called that “pre-existing, not made worse” — probably wrong. Under generations the mismatch also carries a generation mismatch, so lookups that previously limped along on the placeholder entry fail outright, breaking early-boot IPC while leaving the narrower qemu path unaffected. If so, the fix is to make alloc_name honour the requested name — which is exactly C2b, and confirms it cannot be fixed independently of the table.
In XNU a port name is an index into the task’s ipc_space entry table plus a generation counter, and the Mach namespace is entirely separate from the BSD file-descriptor namespace. In this port names are file descriptors, allocated by kern_fdalloc(td, 16, &fd) (ipc_entry.c:453) — lowest-free-at-or-above-16 — and the generation machinery is compiled out (sys/mach/port.h:219, MACH_PORT_GEN(name) is (0)).
Consequence: a name freed by one RPC is handed back bit-identical to the next, and nothing can tell that a cached name now denotes a different object. ipc_object_copyin() then wraps any non-Mach fd into a synthetic port with ip_receiver set and nobody receiving on it, so the send succeeds and the message is swallowed.
| Property | Evidence |
|---|---|
| Permanent wedge, not slowness | 90-second timeout expires unchanged; 5 of 13 sends burned the full 90s |
| Bimodal | sub-second success or never; no intermediate durations |
| Client and server both blocked at once | procstat -kk: client in ipc_mqueue_receive, server thread also parked in ipc_mqueue_receive |
| Traffic-driven decay | 40–60% wedge from cold boot → 0 after ~70–80 sends, then stays 0 |
| Time-independent | 10 minutes idle, zero sends → still 15/30 wedged |
| Resets on daemon restart | warm 0/20 → restart syslogd → 4/20; re-warm → restart notifyd → 10/20 |
The decay is the diagnostic property: two name leaks (§7) push the fd low-water mark monotonically upward, so the recycled band and the live band separate and collisions stop — permanently, because fds never go back down. A daemon restart resets the table to 16 and recreates the dense band.
nextbsd#369 — “Flaky amd64 boot tests: notifyd/syslogd ASL round-trips stall in mach_msg_receive under TCG emulation” — is this bug, and it was closed as emulator flakiness. It is not: it reproduces on real arm64 hardware, survives a 90-second timeout, and has an identified cause in ipc_entry. Emulation only made it easier to hit. That misattribution is why it went unfixed.
The obvious fix — turn the generation counter on — does not work in isolation, and this was verified by writing it. machport_filtops has .f_isfd = 1 (ipc_pset.c:537), so FreeBSD’s kqueue core calls fget(kev.ident) before the filter attaches. libmach registers EVFILT_MACHPORT with the port name as the ident (dispatch_kevent.c:434). Put generation bits in a name and that fget returns EBADF — EVFILT_MACHPORT registration dies, taking libdispatch’s Mach bridge with it.
The written patch is preserved on fix/port-name-generation, committed with a message stating it must not be merged. It becomes correct once Stage 1 lands.
Goal: the filter resolves the port set from the calling task’s ipc_space, with no dependence on the fd table. Names are still fds after Stage 1; nothing may yet change what a name is. Behaviour should be unchanged — this stage only removes the dependency.
| Site | Today | After |
|---|---|---|
ipc_pset.c:537 | .f_isfd = 1 | .f_isfd = 0. This one token flips five kqueue-core behaviours at once: lookup, expand, storage, fdclose, refcount. |
ipc_pset.c:625 | kn->kn_fp = entry->ie_fp; | kn->kn_hook = pset; and retain the reference taken above rather than releasing it. Do not write an fp* into the union slot — with f_isfd = 0 nothing will fdrop it. |
ipc_pset.c:651 (detach) | reaches the pset via kn_fp->f_data->ie_object | pset = kn->kn_hook;, then knlist_remove if still listed, then release the reference on every exit path. |
ipc_pset.c:666 (event) | entry = kn->kn_fp->f_data; — unconditional, before the EV_EOF early-out | seed pset from kn->kn_hook. Must be seeded, not NULL — see R2. |
ipc_entry.c:606 | comment: “we deliberately skip closing the knote so that it will have the last reference to the fp” | False after Stage 1. Pset destruction moves earlier; rewrite the comment. |
ipc_entry.c:675 / :686 | knote_fdclose() reaps the knote on exec | Add explicit teardown. knote_fdclose only walks kq_knlist[fd]; with f_isfd = 0 the knote lives in kq_knhash and can never be found. |
libdispatch needs no changes — verified by sweep: no mach source uses _dispatch_unote_create_with_fd, and the muxnote bucket hash already special-cases EVFILT_MACHPORT. libmach needs no code change either; its ident is already a name, not an fd.
ipc_pset_alloc_name never calls sx_init / knlist_init, unlike ipc_pset_alloc. A pset created that way has an uninitialised ips_note, and knlist_add would write into garbage. Latent today; live after Stage 1. Land the init first, on its own.
ips_reference is released immediately after knlist_add, and the pset survives because kqueue’s fp reference pins it. Remove f_isfd and nothing holds it. The reference must become the knote’s, released in detach. Flipping the flag while leaving the reference as a matched pair moves the write-after-free from a narrow window to the whole knote lifetime — presenting as the same page fault write 0x70.ipc_object_translate only takes io_lock when the caller did not pre-seed *objectp, and io_lock is MTX_DEF. The natural-looking edit (pset = IPS_NULL) panics on the later re-lock. Will not show in a smoke test — needs real message delivery.kqueue_drain runs in whatever thread does the last fdrop of the kq, which for a passed kq is another process. Detach must use the cached pointer and must never re-resolve by name.KNOTE(EV_EOF) sites in ipc_right.c that have never fired at live knotes before.cap_event_rights is replaced by the lookup’s CAP_KQUEUE_EVENT|CAP_KQUEUE_CHANGE.The whole set must land as one atomic commit. .f_isfd = 0 without the event fix is an immediate NULL deref; without the attach-side reference it is a UAF on the first pset destroy.
Confirmed starting state: is_table is not a name-indexed table. It is a chained hash keyed on the object pointer, 7 buckets, and it never grows — ipc_entry_grow_table does not exist in this tree.
Resize the table progression for a pointer array; add a separate reverse-hash array and retarget ipc_hash.c to it; merge the duplicate ipc_entry_hash_delete; delete dead fd_last_used. is_table becomes allocated-but-unused.
Make ipc_entry_lookup require the space lock. Today it takes none, and it ignores its space argument entirely, reading curthread’s fd table (ipc_entry.c:321) — so a task operating on another task’s space silently searches its own. Fixing that is a correctness win in its own right. Do this before the table, while lookups are still cheap. This is where a deadlock surfaces if one exists.
f_countipc_entry_refs() returns entry->ie_fp->f_count (ipc_entry.c:619). A fresh entry has f_count == 1 from the fd-table slot, so urefs and fd references share a counter with different zero points — and the code disagrees with itself about which. Net effect: mach_port_deallocate on a send right takes the > 1 branch, never decrements ip_srights, and two calls are needed to release one right. fork() compounds it: fdcopy() holds every inherited Mach fp, adding +1 per fork per port. ~35 call sites in ipc_right.c, each needing individual review against XNU — not a mechanical substitution.
New ipc_entry_get / alloc / alloc_name / lookup / close / dealloc; a new ipc_entry_grow_table; the new name encoding; per-slot generations. The name encoding, the table, and the lookup have to change together — there is no way to split this further and keep the tree bootable. Compensate with test coverage, not artificial splitting.
Delete ie_fp, mach_fileops, and ~150 lines of copied FreeBSD fd internals. Move mach_port_close’s teardown body somewhere first — do not lose it. Delete the implicit fileport path (§6). Add a per-space entry cap.
Rewrite ipc_entry_list_close, preserving both out-of-tree guards (non-Mach and Linux-ABI processes — each a real crash fix). Decide fork inheritance and exec reset deliberately. This is where daemon-level regressions surface; budget a full bring-up.
Fix libmach/include/mach/port.h, whose macros currently disagree with the kernel’s. Retire the launchd/src/runtime.c workaround that exists because MACH_PORT_INDEX always returns 0. Must land close in time to C2b, not months later — with the new encoding, leaving it in place would size a launchd table by raw name.
Re-enable mach_port_names and mach_port_space_info, both currently disabled and both written assuming a name-indexed table. Recovers what §6 loses.
#define MACH_PORT_INDEX(name) ((name) >> 8)
#define MACH_PORT_GEN(name) (((name) & 0xff) << 24)
#define MACH_PORT_MAKE(index, gen) (((index) << 8) | ((gen) >> 24))The reason the tree deviated — that an index must remain a valid fd — is exactly what Stage 2 removes. Userland already ships Apple’s shape and currently disagrees with the kernel; taking Apple’s layout reconciles them and retires the launchd workaround.
ipc_hash survivesThe indexed table is name → entry; ipc_hash is object → name. Opposite directions; neither subsumes the other. Without it, receiving a second send right to a port you already hold would mint a new name instead of coalescing — and mach_task_self() would return a different name on every call, leaking an entry each time.
ipc_space, yet can still use the parent’s names because fdcopy() duplicated the fds and lookup resolves through curthread’s fd table. Parent and child share ipc_entry structs across two spaces with conflated refcounts. After Stage 2 the child’s namespace is genuinely empty. This must be decided, not discovered.ipc_object_copyin wraps any ordinary fd passed where a port name is expected into a special port with CAP_ALL1 rights — no API, no guard. Structurally impossible after Stage 2. Userland’s fileport_* are already ENOSYS stubs and all consumers handle the failure path, so delete rather than preserve.fstat(2), procstat -f, and lsof. A real introspection loss; C6 is the replacement.RLIMIT_NOFILE implicitly caps port allocation today. Afterwards nothing does, and ipc_entry_alloc_name makes the index user-controlled — an uncapped grow-to-cover is a memory-amplification vector. Add an explicit cap; refuse rather than grow above a threshold.SCM_RIGHTS, and close(2) on one destroys the port. Both stop working.Fixing the two name leaks before Stage 2 makes the system worse, not better. Those leaks — the double-fdalloc in ipc_entry_alloc_name and the f_count/uref off-by-one — are what currently push the fd band upward and drive the failure rate to zero after ~70–80 calls. Restore correct free semantics while names are still ungenerationed and the recycling band stays dense indefinitely: a transient startup wedge becomes a permanent one. It would look like the fix caused a catastrophic regression.
Fold both into Stage 2. Bug (a) is not independently fixable anyway — a correct alloc_name requires a real table.
In-kernel tests via the existing sysctl-handler pattern:
ip_srights decremented. Guards bug (b).mach_task_self() twice must return the same name.Namespace-separation assertions: after boot procstat -f 1 shows no port fds; close(port_name) returns EBADF; an ordinary fd passed as a port name returns KERN_INVALID_NAME.
Acceptance: boot to launchd with syslogd, configd, notifyd and mDNSResponder running, plus a libdispatch workload driving EVFILT_MACHPORT — then the cold-boot wedge measurement (10 blocks × 10 sends) at 0/100. The pre-fix baseline is 35/100, and PR #144 measured 35/100.
Stages 1 and 2 are expected to fix the wedge; that is not demonstrated. The root-cause analysis is verified by reading, but four predicted fixes during this investigation (#126, #129, #130, #144) each addressed a real defect and none moved the measured rate. Treat the outcome as unproven until the acceptance measurement says otherwise.
close, dup, SCM_RIGHTS)? libmach is clean; launchd, libdispatch, CoreFoundation, configd, syslogd and mDNSResponder were not audited. Needs a dedicated sweep before C3.mach_task_fork_bsport suggests full inheritance was tried and panicked.ipc_entry_t across lock drops constantly. Flat-array fidelity would make C1 a correctness prerequisite rather than hygiene.ipc_right_lookup’s lookup-then-lock ordering deliberate (avoiding recursion) or accidental? C1 hinges on it.cap_enter()? Stage 2 removes the only Capsicum check on ports.Both scoping agents were run against a working tree that still had an experimental generation-counter patch applied. Both consequently reported that generations were already live and that EVFILT_MACHPORT was already broken for 63 of every 64 names. That is false for main and for the deployed kernel; it described the uncommitted patch. Each agent flagged the discrepancy in its own open questions, which is how it was caught.
Everything cited here has been re-verified against clean main, and line numbers re-derived. The agents’ findings in files not touched by that patch — ipc_pset.c, ipc_hash.c, ipc_object.c, ipc_right.c, ipc_table.c, task.c — are unaffected. Their independent derivation of the fget(ident) blocker, reached without knowing it had been predicted, is corroboration rather than contamination.
Umbrella: nextbsd-kernel#145.
| # | Stage | Note |
|---|---|---|
| 146 | 1 prereq | ipc_pset_alloc_name init — land first, alone |
| 147 | 1 | EVFILT_MACHPORT resolves through ipc_space — atomic commit |
| 148 | 2 / C0 | reverse hash off is_table — no semantic change |
| 149 | 2 / C1 | lookup takes the space lock, and honours space |
| 150 | 2 / C2a | urefs off f_count — do not land alone |
| 151 | 2 / C2b | the pivot — cannot be split further |
| 152 | 2 / C3 | delete the fd backing |
| 153 | 2 / C4 | fork/exec semantics — where daemon regressions surface |
| 154 | 2 / C5 | userland macros — must land near C2b |
| 155 | 2 / C6 | restore introspection |
| 156 | verify | test harness; acceptance is 0/100 cold-boot |
| 144 | related | bare-port lost wakeup — real, but measured 35/100, identical to pre-fix |