Native automated testing for GNUstep / Gershwin: every category, the options, and the gaps

Output of a multi-agent investigation into GNUstep-native testing plus a live read of the Gershwin ISO test scripts. Every capability claim is grounded in a cited source (§11) or actual repo code. Terminology note: there is no "UIKit" in GNUstep — UIKit is Apple's iOS framework; GNUstep's GUI framework is AppKit (libs-gui). The similarly-named UnitKit is Étoilé's unit-test framework (§2) — a different thing entirely.

1The test-category map — what covers what

First, a plain taxonomy (your categories, lightly standardized — they were close):

The coverage matrix — the core answer. ✔ = strong native fit · ~ = possible/partial · ✗ = not this mechanism.

¹ Apple/XCTest-oriented; GNUstep support unverified — treat as "maybe." ² Only by having the bridged object post synthesized NSEvents (§5). ³ Whole-screen heuristic, blind to individual widgets. ⁴ The current gate uses a colour-count heuristic, not a golden-image diff — true visual regression would add compare (§9).

• The one-sentence shape of the landscape: GNUstep is strong at unit/integration (a real xUnit stack: the Testing framework, UnitKit, XCTest) and has a native functional/integration driver (StepTalk / Distributed Objects that reach into a running app to invoke actions and read state) — but it has no native UI, acceptance, or visual-regression tooling, because it has no working accessibility layer (§4). Those top layers are what you build (§5–§6) or approximate with pixels/OCR.
• Direct answers to your question: UnitKit covers unit tests (xUnit assertions; functional/integration only as far as the code you put in a test body reaches) — it never touches the GUI. StepTalk covers functional/integration by scripting a live app (invoke actions, read object state) — it is a driver, not a unit framework, and does no unit testing.

2Unit tests (native — the strong layer)

• Recommendation for Gershwin components: use the GNUstep Testing framework (make check) — it's the native default, already the regression harness for the core libraries, and needs no extra dependency. Reach for UnitKit only if you want its xUnit ergonomics across an Étoilé-style tree.
• These are in-process: they instantiate objects and assert on them. They never drive a live GUI — that's a different category (§5).

3Integration & functional tests (native driving)

Above the unit line, GNUstep's advantage is that Distributed Objects lets a test process reach into a running app and message its real objects — the native substrate for integration and functional tests. StepTalk is a scripting front-end on top of it.

• Distributed Objects (NSConnection/NSDistantObject) — the transport. An app vends its NSApplication delegate under the app name by convention (the same mechanism NSServices uses), so a harness can rootProxyForConnectionWithRegisteredName:host: in and invoke methods / read state — full semantic control across a process boundary. Category: integration + functional. Gap: you drive whatever the delegate exposes; there's no standard "test object," and no known off-the-shelf DO test harness — you'd build a thin one.
• StepTalk — the scripting driver. A language-independent (default Smalltalk) scripting framework that exposes a live app's object graph to scripts: invoke actions, set/read values, walk the model. Category: functional / integration / lightweight acceptance. It is not a unit framework and does no assertions of its own — you assert inside the script. An app becomes scriptable by shipping a ScriptingInfo.plist; StepTalk can't script an un-instrumented app. Implemented and actively used (gs-desktop treats it as first-class), but niche (latest 0.10.0).
• NSServices — narrow IPC. Invoke a declared service via NSPerformService() and read its pasteboard result. A functional/integration data path for the specific operations an app exposes as services — not a general driver.
• What "functional, no GUI" looks like in practice: spin the component up in a make check test (Testing framework) or drive a running app over DO/StepTalk, then assert on returned state. This is the most robust automated testing available on GNUstep today because it never touches pixels.

4The UI / acceptance wall: no accessibility

Everything above is headless and native. The moment you want to test the GUI, you hit the wall that shapes the rest of this document.

• GNUstep has no working accessibility layer. libs-gui's NSAccessibility.m is ~95% the constant vocabulary (hundreds of role/attribute/notification/action name strings — real and reusable as keys) plus a handful of free functions that are all non-functional: NSAccessibilityPostNotificationWithUserInfo is an empty // FIXME (so no accessibility notifications ever fire) and the rest (UnignoredAncestor/Descendant/Children, RoleDescription, ActionDescription) just return nil. Worse, the per-widget NSAccessibility methods on the AppKit classes are only thinly present (the element classes were added ~0.29.0, largely as skeletons) — so nothing publishes a usable tree or emits change events. There is no AT-SPI2/D-Bus bridge, and Gershwin doesn't want one (§4.1). The AT-SPI ecosystem covers GTK, Java/Swing, Mozilla, OpenOffice; GNUstep is absent.
• Consequence: accessibility-driven tools (LDTP, Dogtail, Accerciser) are blocked — nothing to introspect. And "read the screen after input" becomes a pixel/OCR problem, not a semantic one: you locate controls by coordinates, image template-match, or OCR, and assert the same way. This is upstream in GNUstep, not fixable in the Gershwin ISO tooling — but the in-app-agent approach (§5–§6) routes around it, and §4.1 is the proper native fix.

4.1 · The Gershwin-native fix: NSAccessibility over Distributed Objects (no D-Bus)

AT-SPI2 and D-Bus bridges are explicitly out of scope for Gershwin. AT-SPI2 is the Linux accessibility layer, and its "2" literally means "rewritten to run over D-Bus" — it stands up a separate a11y bus plus a registry daemon (at-spi2-registryd) and marshals every widget into a language-neutral D-Bus Accessible struct. Unwanted here. The correct fix is the one macOS itself uses: NSAccessibility is a Cocoa object API and on macOS does not use D-Bus at all (it rides Apple's own IPC). Gershwin implements that same API and transports it over Distributed Objects — Cocoa-authentic, D-Bus-free, and the same object layer that powers native UI/BDD testing.

• DO is the more natural transport, not a compromise. The accessibility tree is already Objective-C objects — NSView/NSCell/NSControl already carry role, title, value, enabled, and focus. DO vends them (or a thin facade) and a client sends real ObjC messages; you skip the entire marshalling-into-a-neutral-struct step that AT-SPI2 requires.

• One layer, two consumers: a screen reader and a Dogtail-equivalent test/BDD harness consume the same DO-vended tree. The in-app TestAgent (§6) is a client of this — so building it delivers real accessibility as a side effect, the Gershwin way.
• Does not exist yet — a modest greenfield worth building, and less than it sounds because the hardest naming part is already done: NSAccessibility.m already declares the full constant vocabulary (roles/attributes/actions/notifications), so there's no namespace to design — only behavior to add. Two concrete tasks: (1) implement the per-widget attribute/action methods on the AppKit classes — the same view-tree walk the TestAgent already does — and (2) implement NSAccessibilityPostNotification (currently an empty // FIXME no-op) to fan out over DO / NSDistributedNotificationCenter. Then vend the tree over DO and register via gdomap. No external D-Bus dependency — the whole loop is Gershwin apps ↔ Gershwin tools, so there's nothing to shed.

5UI tests — native and external

Two ways to drive real widgets: from inside the app (native NSEvents, semantic) or from outside at the X11/VM layer (toolkit-agnostic, pixel-based).

5.1 · Native, in-process: NSEvent synthesis

• Works: build events with +[NSEvent mouseEventWithType:location:…] / +keyEventWithType:… and post via -[NSApplication postEvent:atStart:] → sendEvent:; they flow through normal hit-testing. Read the result by inspecting NSControl/NSView/model state — semantic, not pixels.
• Limitation: app-level events only — an external process can't post them without being inside the app. So you combine with §3 (DO/StepTalk reach in and post) or run from a test binary hosting the app. This is the seed of the in-app agent in §6.

5.2 · External, toolkit-agnostic (works on GNUstep — input only)

• xdotool (X XTEST extension) is the workhorse: mousemove/click/type/key. Its one semantic hook is search --name for top-level windows (GNUstep sets WM_NAME). Gotcha: window-targeted input uses XSendEvent, which some GNUstep controls ignore as synthetic — prefer global XTEST injection.
• The per-app CI/local harness: Xvfb (headless framebuffer) + the app + xdotool + capture (scrot) + assert (ImageMagick %k/compare, tesseract). SikuliX/PyAutoGUI add "click the thing that looks like this" via OpenCV template-match — the vision stand-in for the missing a11y tree. Add x11vnc to watch failures. Freeze theme/font/DPI or image/OCR matches drift.

6Acceptance / BDD — a native Gherkin framework (proposed)

Human-readable acceptance tests are buildable natively, with direct prior art: Frank and Calabash did this for iOS — Gherkin .feature files on top, an in-app server that walks the view hierarchy underneath. GNUstep suits the same design, and it routes around the §4 accessibility gap:

• An in-app agent IS the accessibility layer, scoped to testing. Running inside the app, it walks the live [NSApp windows]→subview tree and reads the real NSControl/NSView objects — so "click the button titled OK" and "the field should read X" are native and semantic without AT-SPI. That same view-walking code is the natural seed of the DO-native NSAccessibility layer (§4.1) — no D-Bus.

Feature: First-boot login
  Scenario: Admin reaches the desktop
    Given the Gershwin session is running
    When I log in as "admin" with no password
    Then I should see the "System Disk" icon    # agent: findControlByTitle != nil (not an OCR guess)

• Feasibility verdict: runner is trivial (godog/behave are mature); the TestAgent is the real work but modest — a few hundred lines of ObjC for v1 (buttons, fields, menus, windows + a handful of assert verbs). No accessibility required — that's what makes it feasible today. Start with ~6 verbs and grow. Home it in gershwin-developer/Library/Tests/ as the shared, flavor-agnostic acceptance corpus for all ISOs.
• Honest cost: like Frank/Calabash, apps must link the agent in test/dev builds; where the agent isn't present, the black-box fallback keeps the §4 brittleness. StepTalk (§3) could be an alternate backend for apps shipping a ScriptingInfo.plist, but a generic view-walking agent works on any GNUstep app unmodified — the better default.

7System / smoke — the current gershwin-on-nextbsd gate

Today's automation is a system/smoke test: boot the ISO in QEMU, drive input through the monitor socket (no guest agent), capture the framebuffer, assert with a colour-count heuristic + OCR. It lives in .github/workflows/build.yml (test job) + three tests/*.sh scripts.

qemu-system-x86_64 -machine q35 -m 4G -smp 2 $ACCEL -bios "$OVMF" \
  -cdrom "$ISO" -boot d -vga std \
  -serial file:tests/serial.log \
  -monitor unix:tests/mon.sock,server,nowait \   # sendkey + screendump channel
  -nic user,model=e1000 -display none -no-reboot
# ACCEL = "-accel kvm -cpu host" if /dev/kvm else "-accel tcg -cpu qemu64"

mon() { printf '%s\n' "$*" | socat -t2 - "UNIX-CONNECT:tests/mon.sock" >/dev/null 2>&1 || true; }
key() { mon "sendkey $1"; sleep 0.2; }
colors() { identify -format '%k' "$1"; }               # unique-colour count

# boot-test.sh:   PASS when %k > 64        (COLOR_THRESHOLD)  → greeter painted
# loginwindow:    type "admin", ret, ret  (NON-gating, best-effort)
# workspace-test: PASS when %k > 1500 OR tesseract finds "System Disk"  (else exit 1)

• Gating: boot-test and workspace-test have no continue-on-error (they fail the job); loginwindow-test is continue-on-error: true. The release job carries needs: [build, test] + needs.test.result == 'success', so a failed desktop check blocks publish. The passing docs/desktop.png is committed back for the README; boot-artifacts (serial.log, frames, PPMs) upload on every run.
• What it proves: the ISO boots to a graphical greeter, accepts keystrokes, and reaches a painted desktop. What it does not prove: that any specific widget works — it's a whole-screen heuristic (category: system/smoke), and its "regression" power is limited because it's a colour-count, not a golden-image diff (§9).

8VM-level automation (drives §7; how to add clicks)

• Human monitor (today): sendkey, mouse_move (relative), mouse_button, screendump, over a socket via socat. Single-client socket — with libvirt use virsh qemu-monitor-command.
• To add real clicks: monitor mouse_move is relative (can't hit a pixel reliably). Add -device usb-tablet and switch clicks to QMP input-send-event (absolute axis + button + key). Assert with screendump + ImageMagick/tesseract. Stays no-guest-agent — the right layer for whole-ISO tests.

9Regression — cross-cutting

Regression isn't a separate tool; it's running the other categories on every change and catching what came back. Native building blocks:

• Unit/integration regression: the GNUstep Testing framework via make check in CI — exactly how the core libraries guard themselves. Add per-component Tests/ and run on every push.
• Functional regression: the DO/StepTalk or BDD suites re-run each change.
• Visual regression — the current gap/opportunity: the ISO gate uses a colour-count heuristic, which catches "nothing rendered" but not "the wrong thing rendered." True visual regression = ImageMagick compare a screenshot against a committed golden image (per-region to tolerate clocks/cursors), failing on a metric threshold. Cheap to add on top of the existing screendump capture, and it upgrades the gate from smoke to real regression.

10Gaps & a full-pyramid strategy

• Bottom of the pyramid is native and ready — adopt the GNUstep Testing framework (make check) for unit/integration across Gershwin components now; it's the cheapest, most robust win.
• Middle is native driving — use DO/StepTalk for functional/integration of running apps (assert on state, not pixels).
• Top two layers are the gaps — UI, acceptance, and visual regression have no native tooling because of the missing accessibility layer. Fill them with: the in-app TestAgent + Gherkin/BDD (§6, the semantic route), the Xvfb + xdotool + OCR per-app harness (§5.2, the black-box route), and golden-image compare on the ISO gate (§9).
• Everything runs in GitHub Actions and locally from the same sources: make check anywhere; godog/Xvfb per-app in seconds (a shared composite action + a local script); the QEMU gate for whole-ISO. Home the shared suites in gershwin-developer/Library/Tests/.
• The upstream unlock for first-class UI testing (and assistive tech) is a real NSAccessibility implementation transported over Distributed Objects (§4.1) — not an AT-SPI2/D-Bus bridge, which is out of scope for Gershwin. It reuses gdomap as the registry and needs no new bus; the in-app agent is both the near-term substitute and the foundation for it.

11Sources

• GNUstep Testing framework: Implementing test cases · GNUstep test framework write-up · tests-testsuite
• Étoilé UnitKit · swift-corelibs-xctest
• StepTalk · libs-steptalk · StepTalk is alive & well (2023)
• NSConnection / Distributed Objects · NSConnection.m
• libs-gui NSEvent.m · NSAccessibility.m (stubs)
• NSScriptCommand.m (empty stub)
• AT-SPI supported toolkits (GNUstep absent) · AT-SPI2 (D-Bus based)
• DO-native substrate: NSDistributedNotificationCenter · gdomap (GNUstep DO name server) · macOS AXUIElement (accessibility without D-Bus)
• Calabash iOS (Gherkin + embedded server) · Frank (Cucumber + in-app server + UISpec)
• godog (Cucumber for Go) · cucumber/gherkin parser
• xdotool (XTEST) · xdotool man (XTEST vs XSendEvent)
• Xvfb · Xvfb + x11vnc pattern
• ImageMagick compare (visual regression) · Tesseract CLI
• SikuliX · PyAutoGUI · Dogtail (AT-SPI, blocked)
• QEMU Monitor · sendkey/mouse/screendump · QMP input-send-event
• gershwin-on-nextbsd — .github/workflows/build.yml, tests/{boot,loginwindow,workspace}-test.sh (read live from main)

Flagged/unverified: ObjC BDD-unit frameworks (Kiwi/Cedar/Specta) on GNUstep — Apple/XCTest-oriented, compatibility not confirmed. StepTalk 0.10.0 release date (NEWS undated). No existing DO/StepTalk-based GUI-test harness was found (viable, not observed). QEMU mouse_move under -device usb-tablet is version-dependent — the reliable absolute-click path is QMP input-send-event.