mach.ko syscall slot expansion — audit plan

freebsd-launchd-mach has wired 10 Mach syscalls into FreeBSD’s reserved lkmnosys slot range (210–219). Adding an 11th fails with ENFILE. Forward-demand estimate puts total Mach syscalls at 30–40 by the time launchd+configd+notifyd+asl are ported. We need to expand somehow. This is an audit plan, not a design decision — before picking a path we want sub-agents to verify there isn’t a simpler mechanism we’ve missed.

1. Problem statement

FreeBSD’s sys/kern/syscalls.master reserves exactly ten lkmnosys slots at syscall numbers 210–219. The kernel’s kern_syscall_register(..., offset=NO_SYSCALL, ...) scans for slots whose sysent[].sy_call == lkmnosys and grabs the first free one. After ten dynamic registrations, the eleventh call returns ENFILE.

freebsd-launchd-mach hit this boundary at Phase G2b: trying to wire host_set_special_port as the eleventh Mach syscall, kld load printed:

mach: mach_reply_port            registered at syscall 210
mach: task_self_trap             registered at syscall 211
mach: thread_self_trap           registered at syscall 212
mach: host_self_trap             registered at syscall 213
mach: mach_msg_trap              registered at syscall 214
mach: mach_port_allocate         registered at syscall 215
mach: mach_port_deallocate       registered at syscall 216
mach: mach_port_insert_right     registered at syscall 217
mach: task_get_special_port      registered at syscall 218
mach: task_set_special_port      registered at syscall 219
mach: syscall_register(host_set_special_port) failed: 23

(errno 23 = ENFILE.)

Phase G2b shipped a workaround: have libsystem_kernel’s host_set_special_port route through the existing task_set_special_port syscall, with the kernel handler disambiguating by the which argument. That works for one extra entry. It does not scale.

2. Current syscall inventory (10/10)

#	Slot	Name	Phase	User
1	210	mach_reply_port	B Tier 1	libdispatch poller, libxpc, tests
2	211	task_self_trap	B Tier 1	everywhere (mach_task_self())
3	212	thread_self_trap	B Tier 1	tests, libdispatch internals
4	213	host_self_trap	B Tier 1	tests, host_set_special_port shim
5	214	mach_msg_trap	C3	core IPC; libdispatch RECV poller; libxpc; libbootstrap
6	215	mach_port_allocate	F	libxpc, bootstrap server, tests
7	216	mach_port_deallocate	F	libxpc
8	217	mach_port_insert_right	F	libxpc, bootstrap server
9	218	task_get_special_port	G prereq	bootstrap clients via task_get_bootstrap_port
10	219	task_set_special_port	G prereq	tests; bootstrap server (via host_set_special_port routing)

3. Why the 10-slot limit exists

FreeBSD’s syscall table is a fixed-size array at compile time. The kernel allocates one struct sysent per slot at boot; each is ~24 bytes on amd64. The number of slots is determined by SYS_MAXSYSCALL, which is generated from syscalls.master at kernel build time. There’s no runtime knob to grow it — the entire syscall dispatch path (amd64_syscall, fast_syscall_common, etc.) bounds-checks against this compile-time constant.

The 10 lkmnosys entries at 210–219 are a convenience: they let third-party modules grab a slot at kld load time without coordinating with the FreeBSD project for a permanent number. The limit of ten dates to early-1990s BSD when KLDs were uncommon and the typical use case was a single out-of-tree module. The number was never raised because the FreeBSD answer to “I need more” has historically been “commit your syscall to base or use sysctl/ioctl/kern_sysctl” — both of which sidestep the slot question entirely.

So the cap isn’t a hard kernel limit; it’s a convention baked into syscalls.master. The RESERVED slots scattered through 91…296 are a separate convention: “these slot numbers are intentionally left for non-base callers to claim by explicit assignment.” The official comment says “reserved for local or vendor use (not for FreeBSD).”

4. What Apple does instead

Apple’s macOS XNU kernel does not use FreeBSD’s sysent table for Mach traps. There are two separate dispatch tables:

• BSD syscalls (sysent): conventional POSIX-shaped syscalls. Same model as FreeBSD; same compile-time-sized table.
• Mach traps (mach_trap_table in osfmk/kern/mach_trap_table.c): a separate table indexed by negative syscall numbers. The amd64 system_call_64 trampoline branches on the sign of rax: positive routes to sysent, negative routes to mach_trap_table. Around 80 entries today (mach_reply_port, thread_self_trap, task_self_trap, the whole _kernelrpc_mach_port_* family, mach_msg2_trap, etc.).

Userland on macOS uses MACH_TRAP_TABLE_COUNT-aware glue: libsystem_kernel.dylib’s assembly stubs do movq $-N, %rax for trap N, then syscall. The kernel sees the negative number, dispatches accordingly.

So Apple-side, there’s no equivalent of our 10-slot problem: Mach has its own private table with as many entries as Apple wants to define. They simply allocated a different number space.

This is relevant to our Option 4.2 (multiplexer dispatcher): we could approximate Apple’s separation by claiming one FreeBSD syscall slot to host an internally-routed mach_trap_table-style table. The userland shim would dispatch by trap number to the right kernel handler. Agent C’s job below is to verify this is mechanically clean.

5. Risk of using more slots

Three risk axes depending on how we expand:

Mechanism	Risk	Likelihood	Mitigation
Claim a RESERVED slot explicitly (e.g. 91)	A future FreeBSD release assigns slot 91 to a real syscall; our module either fails to load (slot now occupied) or its registration is rejected because sy_call != nosys.	Low historically. The RESERVED slots have been stable for many FreeBSD majors. They’re explicitly documented as off-limits for FreeBSD base.	Pin the module to a known-good OSVERSION range; document the chosen slots in a release manifest; CI rebuilds against new majors before publishing.
Use multiplexer dispatcher in one slot	If FreeBSD ever changes how that one slot is dispatched (unlikely — it’s a single LKM-allocated syscall), our internal trap table breaks. Debug introspection is harder (one syscall name covers many operations).	Very low — we own the dispatch.	None needed at the kernel level. Userland debug helpers (a sysctl mach.trap.<op> tree) preserve introspection.
Patch FreeBSD’s kernel (sys/kern/syscalls.master) to add more lkmnosys entries	Breaks the project’s explicit no-kernel-patches rule; mach.ko would no longer load on stock FreeBSD; ships a forked kernel image; per-major rebuild as syscalls.master drifts upstream.	N/A — disqualified.	—

The dominant risk for RESERVED-slot claiming is a slot collision on a future FreeBSD bump. Looking at FreeBSD’s recent history, the rate of new syscalls is ~5–10 per major release, and they tend to be appended at the high end (currently 602), not into the RESERVED gaps mid-table. The lowest-numbered RESERVED slots (91, 94, 119, 151–153) have been RESERVED since BSD 4.4. So claiming low-numbered RESERVED slots is empirically very safe.

The multiplexer route eliminates this risk entirely at the cost of debuggability.

6. Forward-demand estimate

Phases G2b through eventual launchd-PID-1 plus the cluster of system daemons (configd, notifyd, asl, mDNSResponder, IPConfiguration) require additional Mach syscalls. Non-exhaustive forecast based on Apple/ravynOS source surveys:

Phase	Likely new syscalls	Running total
G2 done (current)	— (host_set_special_port piggybacks)	10
G2 polish	host_get_special_port; possibly mach_port_mod_refs	~12
libxpc port	0 (uses what we have)	~12
bootstrap server / launchd-842 import	mach_port_destroy, mach_port_request_notification (NO_SENDERS, DEAD_NAME), mach_port_get_attributes, mach_port_set_attributes, mach_msg2_trap	~17
configd / notifyd / asl	mach_port_extract_member, mach_port_move_member, mach_port_insert_member, mach_port_construct / destruct, mach_port_extract_right	~22
thread / task introspection	task_threads, task_info, thread_get_state, thread_set_state, thread_resume / suspend	~27
vm	vm_allocate, vm_deallocate, vm_remap, vm_protect	~31
semaphore (existing UNSUPPORTED stubs)	semaphore_signal_trap, semaphore_wait_trap, friends	~36

Rough total: 30–40 Mach syscalls. Some of those can be reasonably multiplexed (e.g. the mach_port_*member family) but not 3–4× over.

7. Candidate paths under consideration

4.1 RESERVED slot expansion

FreeBSD’s syscalls.master contains ~48 entries marked RESERVED, with the explicit comment “RESERVED reserved for local or vendor use (not for FreeBSD).” Slots include 91, 94, 119, 151–153, 159, 167–168, 172, 177–180, 193, 208, 245–246, 249, 258–271, 273, 281–288, 291–296. RESERVED slots have sy_call = nosys.

kern_syscall_register accepts an explicit non-NO_SYSCALL offset and registers if the existing handler is either lkmnosys or nosys — meaning RESERVED slots are claimable, just not auto-allocated.

Pros: stable syscall numbers across reboots; ample headroom (48 + 10 = 58 slots); zero changes to FreeBSD base.

Cons: a future FreeBSD release could co-opt a RESERVED slot, breaking our module. Requires per-syscall explicit slot assignment and tracking.

4.2 Multiplexer dispatcher

One reserved-range slot becomes a generic mach_dispatch_trap(op, args*) that branches on op to call the underlying Mach implementations. All future Mach traps share that one slot; userland shims pack args into a struct.

Pros: only consumes one syscall slot ever; precedent in XNU’s own mach-trap dispatch.

Cons: every new Mach trap needs an op code and arg-packing convention; debugging traces / dtrace become less informative (single syscall covers everything); extra dispatch indirection per call.

4.3 Per-syscall routing (status-quo workaround scaled up)

Keep finding ways to overload existing syscalls’ which / op / disposition arguments to handle new operations. Phase G2b does this for host_set_special_port via task_set_special_port.

Pros: no new infrastructure.

Cons: doesn’t scale; quickly creates obscure API overloading; each new operation requires ad-hoc disambiguation rules; not viable for 20+ more syscalls.

4.4 Patch FreeBSD’s kernel (specifically sys/kern/syscalls.master)

Add more lkmnosys entries to syscalls.master. The file lives at sys/kern/syscalls.master and is compiled (via tools/makesyscalls.lua) into sys/kern/init_sysent.c, which is linked into the kernel. Modifying it requires rebuilding the kernel and shipping the modified kernel image — this is a kernel patch specifically, not a base-system patch in the broader sense. (The project freely patches other base components: gershwin libdispatch fixes, install-layout adjustments, ldconfig drop-ins, etc.) The hard constraint is the kernel staying stock so mach.ko remains an out-of-tree loadable module on every supported FreeBSD release.

Pros: most idiomatic from FreeBSD’s perspective if upstreamed.

Cons: violates the project’s explicit no-kernel-patches rule (mach.ko must build/load on stock FreeBSD); ships a forked kernel; per-major rework as syscalls.master drifts upstream. Disqualified.

8. Audit plan — what sub-agents will verify

Before picking a path we want concrete answers to the following. Each bullet maps to one sub-agent dispatch.

Agent A — FreeBSD syscall-extension mechanisms

Read sys/kern/init_sysent.c, sys/kern/kern_syscalls.c (or wherever kern_syscall_register / kern_syscall_helper_register lives), and adjacent infrastructure. Confirm:

• Does kern_syscall_register with an explicit offset accept RESERVED slots (where sysent[N].sy_call == nosys)? Cite the exact source location.
• Is there a second syscall table for KLDs (separate from the global sysent) that we’ve missed — e.g. kern_syscall_helper_register, module_register_init, freebsd32 / linuxulator-style alternate tables?
• Are there sysctls / kenv tunables that change SYS_MAXSYSCALL or alter the syscall-table sizing? Confirm one way or the other.
• What happens if a KLD reserves an explicit offset, then a future FreeBSD release assigns that slot to a real syscall? Loader behavior? Boot fault? Silent override?
• Are there NODEF / NOTSTATIC or other flag combinations in syscalls.master that allow append-style growth at module-load time?

Agent B — survey existing Mach-syscall users

Across nextbsd/, ravynos/, and the local freebsd-launchd-mach tree, produce a complete frequency table of Mach syscalls actually called. We want hard numbers, not estimates.

• For each Mach syscall (mach_port_*, task_*, thread_*, host_*, mach_msg_*, semaphore_*, vm_*), list the consumers (launchd, libxpc, configd, notifyd, asl, etc.).
• Identify syscalls that are theoretically in the Mach surface but never actually invoked by any consumer in our porting scope. Those don’t need slots.
• Identify syscalls that are only called via MIG (so they go through mach_msg, not direct syscalls). Those don’t need slots either.
• Output: realistic minimum count of direct-syscall Mach traps we’d need to ship a working stack. Replace the 30–40 estimate with a measured number.

Agent C — XNU’s own Mach-trap dispatch

How does Apple’s actual XNU expose Mach traps? Mach traps are not regular syscalls on macOS — they live in a separate table (mach_trap_table) with negative syscall numbers in some configurations. Survey:

• Does XNU’s model give us any inspiration for how to expose Mach traps in a separate space from the FreeBSD sysent table?
• Is there a way to register a custom syscall handler (single entry point) and route Mach traps internally based on the trap number, analogous to XNU’s mach_trap_table dispatch?
• If yes, the answer to our slot problem might be “one FreeBSD syscall slot, hosting an XNU-style mach-trap-table internally,” which is mechanically similar to option 4.2 but with the architectural justification XNU provides.

Agent D — impact on existing wired syscalls if we change the scheme

Whatever we pick (RESERVED slots, multiplexer, hybrid), we’d want to migrate the existing 10 syscalls into the new scheme so the implementation is uniform. Audit:

• What breaks if syscall numbers change (e.g. moving mach_reply_port from auto-allocated 210 to RESERVED slot 91)? Are any consumers hard-coding numbers, or do all of them resolve via sysctl mach.syscall.<name>?
• Does the sysctl-resolution path work uniformly for multiplexer-style dispatch (where the “number” might be an op-code inside a single syscall)?
• Does the smoke-test harness need adjustments? Boot-test expect grammar?

9. Decision rubric

Once Agents A–D return, evaluate each candidate path against:

Criterion	Weight	Why it matters
Respects no-FreeBSD-kernel-patch constraint	Hard	Project rule. Disqualifying if violated.
Headroom for 30–40 future Mach syscalls (or whatever Agent B finds)	High	Phase H+ won’t land otherwise.
Survives FreeBSD-base bumps without per-major rework	High	Project ships per-major release artifact.
Debuggability: traces / dtrace / ktrace remain informative	Medium	Bug hunt cycles matter; we’ve already spent N CI iterations finding subtle Mach bugs.
Apple-canonical surface preserved (consumer code unmodified)	Medium	libxpc, future launchd-842 import must compile without surface rewrites.
Code complexity / new infrastructure	Low	Smaller is better when other criteria tie.

10. Next step

Joe reviews this plan. On approval:

1. Dispatch Agents A–D in parallel (each returns a focused report under ~600 words).
2. Synthesize findings into a follow-up “decision” doc that picks one path with rationale.
3. Land the chosen migration as a discrete commit before Phase G2c (standalone bootstrap daemon) so future Mach syscalls drop in cleanly.

Phase G2b’s routing workaround stays in place in the meantime so the build remains green; the migration commit revisits it.

Spike doc, 2026-05-13. Captures state at freebsd-launchd-mach@main commit 2e3fc7e (Phase G2b post-routing-fix).