← Back · relates to nextbsd #278 (pkg/ports lib resolution), #277 (path_helper), and the live-root design
/usr/local/lib without a boot serviceNextBSD runs the stock FreeBSD ports tree (pkg) on an Apple-shaped system — launchd PID 1, Apple userland, no rc.d. pkg binaries link bare sonames (libpcre2-8.so.0) and rely on the runtime linker to find them in /usr/local/lib — but NextBSD registers no such path, so git dies at exec: ld-elf.so.1: Shared object "libpcre2-8.so.0" not found. The reflexive fix — a boot-time ldconfig one-shot — is exactly the rc.d pattern we're leaving behind, and it turns out it is not how Apple does it either. This doc maps the real solution space from three angles: FreeBSD's rtld internals, NextBSD's current build, and Apple's dyld model.
Recommendation in three moves
/usr/local/lib into rtld's compiled default search path (Lever B) — a one-line build override in the nextbsd-freebsd-compat rtld build (-DSTANDARD_LIBRARY_PATH="/lib:/usr/lib:/usr/local/lib"). No file, no daemon, no boot step — and it is the only lever that also covers setuid/setgid binaries. Fixes the reported failure outright.WatchPaths daemon (no RunAtLoad) over /usr/local/libdata/ldconfig runs ldconfig into a persistent path when a port adds/removes a lib dir; rtld reads it via a globally-set LD_ELF_HINTS_PATH. The hints live in the writable layer (tmpfs upper / disk), never baked. This is the FreeBSD analog of update_dyld_shared_cache: Apple rebuilds its cache at install time, not at boot.-Wl,-rpath,/usr/lib/system convention NextBSD already uses is the Apple model. That's the north star; it just can't be retrofitted onto the stock ports tree.Explicitly rejected: a boot-time ldconfig one-shot LaunchDaemon. Apple has no equivalent, it's the rc.d pattern, and it's unnecessary given the above.
Locked after review what the back-and-forth settled:
build.sh generating the initial /var/db/ld-elf.so.hints from the base's lib dirs (the existing ldconfig prime, redirected to the persistent path and extended). It's a build step, not a boot step. Lever B (compiled default) remains primary and is the only setuid-safe lever; the prebuilt baseline gives LD_ELF_HINTS_PATH a valid target from boot one and covers any base lib dir beyond the compiled default; the WatchPaths daemon keeps it fresh on package change. Three moments — compile / image-build / package-change — mapping to the linker built-in / shipped cache / update_dyld_shared_cache./usr/local/libdata/ldconfig/ on a live box already holds gcc14 → /usr/local/lib/gcc14, perl5 → /usr/local/lib/perl5/5.42/mach/CORE, pkg → /usr/local/lib/compat/pkg — version-specific subdirs Lever B can't hardcode. One WatchPaths on that directory covers all present and future ports.WatchPaths/QueueDirectories and has the EVFILT_VNODE machinery, so a WatchPaths daemon (no RunAtLoad, with ThrottleInterval) is the path-watch trigger — cleaner than a pkg trigger and decoupled from pkg. (A pkg trigger remains a viable fallback if the WatchPaths job-activation path turns out not to be fully ported.)/System/Library/LaunchDaemons found NextBSD runs zero boot one-shots today — every daemon is persistent (KeepAlive) or on-demand (syslogd/aslmanager are socket/trigger-activated, not boot scans). A boot ldconfig would be the lone exception.cleanvar/cleartmp wipes /var at boot, and nothing tmpfs-mounts over it. The generated hints persist on disk (installed) and track the ephemeral session in lockstep with the pkg installs themselves (live ISO).A FreeBSD ports binary records each dependency as a bare DT_NEEDED soname — libpcre2-8.so.0 — with no path. It is not self-describing; it trusts the system to know where /usr/local/lib is. On stock FreeBSD that knowledge comes from the ld-elf.so.hints file that rc.d/ldconfig rebuilds at every boot. NextBSD has no rc.d and never builds those hints, so the runtime linker only searches its compiled-in default (/lib:/usr/lib) and the lookup fails.
Observed on a booted T460s: ldd git shows libpcre2-8.so.0 => not found and libintl.so.8 => not found — both are installed in /usr/local/lib — while the base libs in /lib resolve fine. The one-line confirmation: ldconfig -m /usr/local/lib makes git --version run. The question is not whether to point rtld at /usr/local/lib, but how — without a boot-time scan.
macOS does not use ld-elf; it uses dyld, which has no ldconfig, no hints file, and no ld.so.cache. Resolution is performed in-process at exec/dlopen from paths baked into each Mach-O:
LC_ID_DYLIB) — every dylib carries its own canonical path; the static linker copies it verbatim into each client as an LC_LOAD_DYLIB. If it's an absolute path, dyld walks straight to it — no search, no registry.@rpath / @loader_path / @executable_path — placeholders resolved against the client's LC_RPATH entries (or the loader/executable directory). The ELF analog of @loader_path is $ORIGIN.The dyld shared cache is the closest thing to "a service," but it holds OS libraries only, is built by the installer (update_dyld_shared_cache) at install/update time (not boot, and only effective after reboot), and is never a registry for third-party libraries. Homebrew (/usr/local or /opt/homebrew) and MacPorts (/opt/local) resolve their libs by rewriting install names to absolute prefix paths at package/pour time with install_name_tool (Homebrew's keg_relocate). The binaries are made self-describing; nothing scans the disk.
Takeaway A dislike of boot one-shots matches Apple exactly. macOS has zero boot-time linker indexing — no daemon is consulted for routine resolution at all. The only "cache" is an OS-image artifact rebuilt by the installer, never at boot, never for third-party libs. So the Apple-faithful instinct is: self-describing binaries, fixed at package time; any global cache is an event-driven, install-time, persistent artifact — not a boot step.
For a bare soname (driver find_library(), rtld.c:1915), rtld searches in order (rtld.c:1969–2005): DT_RPATH → LD_LIBRARY_PATH → DT_RUNPATH → LD_LIBRARY_PATH_FDS → the hints file (rtld.c:1996) → the compiled standard path (rtld.c:2001). With no rpath/runpath and no LD_LIBRARY_PATH, only the last two can save a pkg binary. Four levers fall out:
LD_ELF_HINTS_PATHThe hints path defaults to /var/run/ld-elf.so.hints (rtld_paths.h:48–54) but is overridable at runtime by the LD_ELF_HINTS_PATH environment variable (rtld.c:385, read at 756, opened at 2148). Set it globally (e.g. in launchd's PID 1 environment, inherited by every child) to a non-tmpfs path and rtld reads persistent hints. caveat it's flagged unsecure and is purged for setuid/setgid binaries (rtld.c:385,738–743), which then fall back to the compiled default path.
/usr/local/lib into the compiled default pathSTANDARD_LIBRARY_PATH is "/lib:/usr/lib" (rtld_paths.h:76–78) and is guarded by #ifndef — so it is overridable by a -DSTANDARD_LIBRARY_PATH=... in CFLAGS at build time, with no source edit. Baking "/lib:/usr/lib:/usr/local/lib" resolves the bare-soname case at search step 7 with zero hints, zero service, and it covers setuid binaries too. The most robust single lever.
$ORIGIN / RUNPATH (the self-describing route)rtld supports $ORIGIN-relative runpaths (rtld.c:1251,1264) — the FreeBSD analog of Apple's @loader_path. This is the genuinely Apple-native answer, but it requires every binary to carry the runpath; the stock ports tree links bare sonames by design, so it can't be retrofitted wholesale (and $ORIGIN is also disabled for setuid). Right for first-party binaries only — which NextBSD already does (§4).
The hints format is a position-independent header plus a single colon-separated dir string — no inodes, timestamps, or host state (elfhints.c:266–319; elf-hints.h:35–46), little-endian by default. So a hints file is bakeable and cross-boot stable, and ldconfig can be told to write any path. Run it from a pkg install/deinstall trigger to a persistent file; pair with lever (a) so rtld reads it. Event-driven, persisted, no boot one-shot — the structural twin of update_dyld_shared_cache.
For completeness, stock FreeBSD's rc.d/ldconfig default ldconfig_paths is /usr/lib/compat ${localbase}/lib ${localbase}/lib/compat/pkg and ldconfig_local_dirs=${localbase}/libdata/ldconfig (the per-package drop-in convention) — the dir set any replacement must reproduce.
-Wl,-rpath,/usr/lib/system (dozens of sites in build.sh). build.sh:748–758 states the philosophy outright: hints are primed only for /usr/lib/system because "our binaries … all bake -Wl,-rpath,/usr/lib/system at link time, matching Apple's macOS convention where LC_RPATH load commands make each binary self-describing and ld-elf.so.conf-style global registries aren't used." This is Lever C, already shipped./usr/local/lib. chroot rootfs ldconfig -m /usr/lib /usr/lib/system (re-run after each Apple-lib install) builds hints covering only those two dirs — which is exactly why ports binaries fail./var/run is not tmpfs. It is a plain baked directory; nothing — not launchd, not fstab, not the live union — mounts a tmpfs over it. A build-baked hints file therefore survives reboot on both the installed UFS image and the live uzip union. to confirm yet the tested ISO had no /var/run/ld-elf.so.hints at runtime — so either the priming isn't reaching the uzip or it fails silently (§9). Either way it would never have listed /usr/local/lib.nextbsd-base artifact built by nextbsd-freebsd-compat's from-source /usr/src build. So Lever B's -D override lives there — not in nextbsd-kernel (kernel-only src-overlay) and never in freebsd-src.launchd_root_make_writable() (launchd.c:424, called at 313), which runs after jobmgr_init() and before launchd_scan_launchdaemons() (launchd.c:352) — i.e. root is rw before any daemon. It only remounts /; it touches no tmpfs and no /var/run. (This is where an "after root rw" step would anchor, if we were doing one — we're not.)/etc ships none of ld-elf.so.conf, shells, man.conf, manpath.config, profile, csh.cshrc (overlay has only asl.conf, fstab, group, hosts, login.conf, master.passwd, motd.template, nsswitch.conf, pam.d/, ssh/). The missing config is tracked in #278.| Option | Verdict | Apple-alignment | Covers setuid? | Boot step? |
|---|---|---|---|---|
Boot-time ldconfig LaunchDaemon (one-shot) | reject | None — Apple has no boot linker step | n/a | yes (the rc.d pattern) |
B — compiled default incl. /usr/local/lib | primary | Neutral (a sane default; no registry) | yes | no |
D+A — persistent hints, rebuilt on pkg change, read via LD_ELF_HINTS_PATH | secondary (the tail) | Strong — twin of update_dyld_shared_cache (install-time, persistent) | no (env purged) | no (event-driven) |
C — self-describing first-party binaries (rpath/$ORIGIN) | already shipped | Exact — the dyld model | no | no |
| Self-describe the whole ports tree (rpath fixup at pkg time) | north star, impractical | Exact — Homebrew keg_relocate | no | no |
| Static baked hints only (no rebuild trigger) | half-measure | Weak | no | no, but stale on user pkg installs |
NextBSD ships the stock ports tree, so the pure-Apple answer (self-describe every binary) isn't retrofittable. The faithful-and-pragmatic shape is a layered one, none of it a boot service:
/usr/local/lib (and /usr/lib/system) into rtld's compiled default in nextbsd-freebsd-compat. Config-free, boot-free, covers setuid, survives the union and tmpfs entirely. This alone fixes the reported libpcre2-8.so.0 failure for the common case (libs in /usr/local/lib)./var/db/ld-elf.so.hints) covering the full ports dir set (/usr/local/lib/compat/pkg + every libdata/ldconfig drop-in); rtld reads it via LD_ELF_HINTS_PATH set once in launchd's PID 1 environment. This is the macOS analogy made literal: pkg is the "installer," the trigger is update_dyld_shared_cache, the file is the shared cache — rebuilt on package change, persisted, never at boot.-Wl,-rpath (and $ORIGIN where relative) — the existing convention, which is the truest Apple model.In practice Lever B may be all most users ever need (almost every ports lib lands in /usr/local/lib); D+A is the completeness layer for packages that scatter libs elsewhere. Crucially, the user's instinct is vindicated twice over: a boot one-shot is both un-Apple and unnecessary.
Both the LD_ELF_HINTS_PATH env lever (a) and $ORIGIN runpaths (c) are purged for setuid/setgid binaries (rtld.c:738–743, 1269) — a tainted binary ignores them and falls back to the compiled default path. So any setuid pkg binary that needs a ports lib can only be served by Lever B (the compiled default) or by a real file at the literal compiled hints path. This is the decisive reason B is the non-negotiable foundation and D+A is an augmentation, not a substitute.
libexec/rtld-elf — CFLAGS+= -DSTANDARD_LIBRARY_PATH=\"/lib:/usr/lib:/usr/lib/system:/usr/local/lib\" (via a Makefile fragment / src-overlay, not a freebsd-src edit). Verify with ld-elf.so.1 -v ("Default lib path")./usr/local/etc/pkg/triggers/ or a base @postexec) that runs ldconfig -f /var/db/ld-elf.so.hints <dir set> on install/deinstall. The dir set = /usr/lib /usr/lib/system /usr/lib/compat /usr/local/lib + expansion of /usr/local/libdata/ldconfig/*.LD_ELF_HINTS_PATH=/var/db/ld-elf.so.hints in launchd's PID 1 environment so all children inherit it (vendored launchd, or launchctl setenv very early). Confirm it reaches ssh/login sessions./etc/ld-elf.so.conf (listing /usr/local/lib) in overlays/etc/ as the canonical ldconfig input, alongside the other missing base /etc files from #278./var/run hints are absent at runtime and make the build prime /usr/local/lib too, or drop the priming once Lever B lands and it's redundant./var/run/ld-elf.so.hints absent on the tested ISO when build.sh primes it and /var/run isn't tmpfs? (Priming not reaching the uzip? ldconfig -m failing in the chroot with no pre-existing hints? A clean step? Or the tested ISO predates the priming.) Resolve before trusting any baked-hints path.LD_ELF_HINTS_PATH set in launchd's PID 1 env actually reach all children — getty/login shells and ssh sessions — given the same propagation quirks we hit with PATH (#277)?stat() of /usr/local/lib on every search miss. Almost certainly negligible, but worth a sanity check on a daemon-heavy boot./usr/lib/compat / /usr/local/lib/compat/pkg in the compiled default, or only in the event-driven hints?freebsd-src@releng/15.0: libexec/rtld-elf/rtld.c (search order 1969–2005; LD_ELF_HINTS_PATH 385/756/2148; $ORIGIN 1251/1264; setuid purge 738–743/1269), libexec/rtld-elf/rtld_paths.h (_PATH_ELF_HINTS 48–54; STANDARD_LIBRARY_PATH 76–78), sbin/ldconfig/elfhints.c (hints writer 266–319), include/elf-hints.h (format 35–46), libexec/rc/rc.d/ldconfig + libexec/rc/rc.conf (defaults).
nextbsd: build.sh:748–758 (priming + rpath philosophy), the -Wl,-rpath,/usr/lib/system sites, overlays/etc/fstab, src/launchd/src/launchd.c:313,424 (make-writable + ordering); rtld/ldconfig sourced from nextbsd-freebsd-compat.
Apple: Run-Path Dependent Libraries, dyld(1), install_name_tool(1), dyld_shared_cache, Homebrew keg_relocate.
Spike for nextbsd #278. Investigations: FreeBSD rtld/ldconfig, NextBSD build/overlay/launchd, Apple dyld — run in parallel and synthesized here. The throughline: NextBSD already does the Apple thing for its own binaries (self-describing rpath); the gap is only the stock ports tree, and the fix is a compiled default + an install-time persistent cache — never a boot service.