← Back · part of NextBSD Research · sibling of the virtio-gpu DRM plan · decision record: arm64 KMS options · console policy: quiet console plan

NextBSD under Apple Virtualization.framework — boot, console, and the fbdev gap

Apple’s VM stack is the one arm64 target where NextBSD has never booted, and the reason turned out not to be ACPI, GIC, or graphics. It was USB. This page records what was measured on real hardware, what the remaining blockers are, who else is already working on them, and the order to fix them in.

Investigated 2026-08-18, updated 2026-08-19 on an Apple Silicon Mac, macOS 26.5.2. Every claim tagged measured was observed directly on that machine against a NextBSD arm64 install (kernel 20260818-174943) and the NextBSD-arm64-20260817-065847 ISO. Upstream claims carry source links. Claims from binary analysis of Apple’s framework are tagged as such and are the least durable thing here.

1. What was measured

A NextBSD arm64 install was copied out of its UTM qcow2 to a sparse raw image and booted under Virtualization.framework through a ~90-line Swift harness, ad-hoc signed with com.apple.security.virtualization. Everything below is first-hand.

The crash/hang split is diagnostic, not cosmetic: a crash means the guest touched a device model Apple refuses; an idle guest means the kernel stopped scheduling; a spin means it is looping. The triggered thread name in ~/Library/Logs/DiagnosticReports/com.apple.Virtualization.VirtualMachine-*.ips discriminates further — thread.cpu-N is a guest-triggered device assertion, virtual-machine-service is a configuration the VMM refused at start.

2. The xHCI fatality

The crash report’s faulting function, disassembled, references only xHCI MMIO-decode strings:

"Security Assertion Hit: "
"Guest attempted to write into RsvdP in the Host Controller Runtime Registers's
 Interrupter Register Set \"%u\". "
"Guest attempted to write into a Host Controller Operational Registers' port
 register at offset \"0x%x\". The offset is past MaxPorts of this device."
"Unhandled endpoint type %d"   "Non-zero doorbell target %u is unexpected"

The call site is adrp x1, "Report an error"; bl …, and that callee ends in adrp x0, "assertion failed"; bl __os_crash; brk #0x1. This is a deliberate design choice: an unmodelled guest write kills the VMM rather than being ignored.

It also matches the only known-good FreeBSD-on-VZ writeup independently, whose blocker #4 reads verbatim “USB/XHCI is fatal — any USB device makes VZ present an XHCI controller and FreeBSD’s USB stack dies”.

Sources: local crash reports 2026-08-18; networkextension/FreeBSD-vz.

3. Why the guest is mute — three routes to a console

VZ gives an arm64 guest no console path that FreeBSD can currently use. This is a platform fact, not a bug in our configuration:

• No GOP — measured. So vt_efifb can never attach, and there is no MODINFOMD_EFI_FB to hand over. The fallback graphics route does not exist on this platform.
• No SPCR, no DBG2, no UART in the DSDT — a real VZ Linux guest shows exactly five tables (RSDP/XSDT/FACP/DSDT/GTDT/APIC/MCFG, OEM APPLE, creator Apple Vz). SPCR is generated per configured device, and the public API configures none that qualify. Note FreeBSD ignores DBG2 anyway unless hw.acpi.enable_dbg2=yes, and SPCR structurally cannot describe a virtio console — its Interface Type byte defers to DBG2 serial subtypes, all of which are UART variants.
• virtio-console is tty-only in FreeBSD — sys/dev/virtio/console/virtio_console.c creates /dev/ttyV0.x but registers no consdev/cnputc, so it cannot carry printf from early boot. Linux uses the same device as hvc0, which is why every VZ tool’s “serial output” works for Linux guests and not for us — including swift-vm-cli, which wires VZVirtioConsoleDeviceSerialPortConfiguration to stdio in raw mode.

hw.uart.console="mm:0x20010000,br:115200,rs:2,rw:4,xo:0"

4. Remaining boot blockers, ranked

With xHCI removed, the guest reaches the kernel and goes idle. These are the candidates, ranked by how well they fit that signature, each with the cheapest test. Status reflects what has since been measured.

Note also that bus_probe() in sys/arm64/arm64/machdep.c has no health check and no fallback — the presence of an RSDP alone wins, even if the tables that follow are unusable. And FreeBSD detects hypervisors only via SMBIOS, so under VZ vm_guest is unset and no guest quirks apply (bug 292082 / D54595 proposes boot_el-based detection).

5. Prior art — we are not first

The platform, as measured by others

Worth recording because it differs from QEMU’s virt in ways that break assumptions: RAM is at 0x7000_0000, not 0x4000_0000 (PCIe ECAM sits there instead); GICv3 only, no ITS and no MSI/MSI-X — PCIe is legacy INTx on SPIs; PSCI 0.2 with HVC conduit and an EL1-only guest, so SMC faults; virtio-PCI modern only, no virtio-mmio; 4K or 16K granule, never 64K. Our own acpi.rsdp = 0xefbf0018 is consistent with the 0x70000000 RAM base and inconsistent with a QEMU-virt layout — a useful independent confirmation.

6. The fbdev gap — closed

This closes the open question left dangling in the graphics plan §10 (“Did drm-kmod’s core build with CONFIG_DRM_FBDEV_EMULATION?”) and corrects the rationale currently written into graphics/drm_shmem_helpers/Makefile.

Enabling drm_fbdev_shmem.c in CI produced exactly 8 errors, no missing headers, no incomplete types: fb_deferred_io_init/_cleanup undeclared, fb_WARN_ON_ONCE undeclared, and fbdefio missing from struct drm_fb_helper and struct linux_fb_info.

The console’s damage path is separate and already implemented and exported: vt_drmfb draws only through info->fbops->fb_imageblit → drm_fb_helper_damage_area() → fb_dirty → our drm_atomic_helper_dirtyfb → virtio_gpu_cmd_transfer_to_host_2d() + resource_flush. vt(4) already batches at 25 Hz (VT_TIMERFREQ) — the same order as Linux’s HZ/20 defio delay, with exact damage clips rather than full-screen blasts.

So the fix is ~20–40 LOC of stubs, with upstream precedent: drm-kmod commit 2ac98d2787 (wulf@, 2026-06-14, on master but not yet 6.12-lts) adds the empty fb_deferred_io_cleanup() and drops the __linux__ guard; Phabricator D57488 moves this code into base keeping the stubs. Implementing real deferred I/O would fight the direction of travel and needs VM plumbing FreeBSD lacks (vm_page has no lru).

Three runtime traps the compiler will not catch: drm_fb_helper_deferred_io is declared but __linux__-guarded out (undefined symbol at load — tools/check-kext-symbols.py catches it); fb_deferred_io_mmap() is a live panic(); and shmem’s fix.smem_start == 0 means vt_fb_mmap() would hand out pages nothing ever flushes, so set FB_FLAG_NOMMAP.

Scope note: this unlocks shmem-backed fbdev only — i.e. virtio-gpu, plus the other shmem consumers (ast, mgag200, cirrus, hyperv) our drm_shmem_helpers module already pays for. Bochs and VBox are unaffected: they are TTM-backed and already work.

OpenBSD’s viogpu.c is the counter-example worth knowing: it transfers and flushes the whole screen every 10 ms with no damage tracking, from a non-DRM wsdisplay driver. It proves a crude periodic flush is viable, and simultaneously that FreeBSD’s vt_drmfb path is the better of the two.

7. Reconciling with V7 and Option D

The virtio-gpu plan carries standing constraint V7: never add device virtio_gpu to the NEXTBSD kernel config, and arm64 KMS options §Option D recommends nodevice virtio_gpu, because base’s driver registers at VD_PRIORITY_GENERIC+10 against efifb’s +1 and displaces the console that works.

Restated constraint: “never rely on base virtio_gpu(4) as the final console, but do keep it as the first one, and hand the device over explicitly once the KMS kext is resident.”

The handover mechanism, since newbus will not do it for us. It remains true that there is no re-bid — bus_generic_driver_added() re-probes only DS_NOTPRESENT children, and releng/15.1 has no DF_REBID — so an attached vtgpu0 is never displaced by a better driver arriving later. Two pieces close that:

• kernel#76 teaches the in-kernel IOKit present-scan to look through virtio_pci to a base-shadowed virtio-gpu, exactly as it already looks through vgapci to an unbound drmn. Without it nothing ever asks kextd for the bundle: device_nomatch does not fire for a device base matched perfectly well.
• kernel-modules#39 makes the kext take the device itself on load — device_detach() then device_probe_and_attach(), inside one bus_topo_lock() section, on a deferred task. That is DEV_SET_DRIVER’s sequence written in exported functions, and BUS_PROBE_VENDOR (kernel-modules#37) is what wins the resulting bid.

Result, on a stock arm64 boot with nothing typed: base paints the console from VT: initialize with new VT driver "virtio_gpu", kextd loads the bundle the scan asked for, the kext takes the device, and vt switches to drmfb — console the whole way through, plus card0. This is the shape VZ will inherit for free once it boots at all.

8. Plan

vtgpu0: <VirtIO GPU> on virtio_pci1
VT: initialize with new VT driver "virtio_gpu".
vtgpu0: detached
virtio_gpu_drm0: <VirtIO GPU (DRM/KMS)> on virtio_pci1
[drm] Initialized virtio_gpu 0.1.0 for virtio_pci1 on minor 0
VT: Replacing driver "virtio_gpu" with new "drmfb".
/dev/dri/card0  /dev/dri/renderD128

9. Open calls

1. Do we carry the four virtio commits locally, or wait for an MFC to stable/15? They are merged to main only.
2. Does virtio_console join the compiled-in tier in driver delivery §3, or take a documented exception?
3. Contact networkextension and tjfontaine before writing code — two parties are already on this platform, and one has it booting. Given that stock main spins (§5), their patch set is the dependency; the question is whether we carry it, or help push it upstream.
4. Is the OpenBSD trick (synthesize FDT from ACPI in the loader) worth adopting as a bring-up shortcut, given bus_probe() has no fallback when ACPI is present but unusable?
5. Does the harness belong in swift-vm-cli (which already does the VZ + virtio-console half) rather than living as a scratch tool?
6. Restate V7 / Option D per §7, or leave them and add the qualification only here? Answered 2026-08-19: both are withdrawn for virtio-gpu — base stays compiled in and the kext takes the device (§7). The peer pages still need editing to match.

Sub-plan of the NextBSD graphics plan where it touches the console; a peer of the virtio-gpu DRM plan and the arm64 KMS decision record, which it amends in §7. Everything tagged measured was observed on 2026-08-18 (VZ, macOS 26.5.2) or 2026-08-19 (arm64 qemu guest, kernel 20260819-184420) against NextBSD arm64; upstream claims are linked to source. Claims derived from disassembling Apple’s framework are marked and are the least durable material here — they describe private, undocumented API and may change without notice.