Testing and the E2E scenario suite

Layers, per the design spec's testing strategy:

• Unit tests live beside their packages and run with make test (race detector on). TEST_PKGS narrows it to some packages and TEST_SKIP leaves some out. CI's build-and-test job runs the whole suite. A test that needs a database outside internal/store opens it with storetest.Open (internal/store/storetest), which starts a new file as a copy of one migrated once per test binary: applying every migration costs seconds per database under the race detector. The scripts job runs make test-scripts and make test-native-hooks, and lint runs the advisory make vulncheck. Permission matrices, budget math, configuration upload and file revision rules, tailnet auth edge cases, scheduler transitions, and the local gateway's own behaviors are proven there, once, and the E2E suite does not restate them. Role changes belong to the same layer: internal/sshd/role_test.go and internal/sshd/permissions_test.go own promotion, demotion, the last-admin guard and what each role may do, and the SPA's half of it is in its rendered route tests. The multi-member E2E row below joins members and administers them; it does not re-prove the matrix. internal/fallback owns usage-fallback selection against a stub quota reader and a real store: the 95% threshold, automatic and explicit order, unreadable usage, reset windows, a revoked share, and the empty pool. The mid-run move is proven in internal/scheduler/usage_fallback_test.go with the fake runtime and acpmock: an Enhanced turn failure, a headless Claude exit and a Standard StopFailure report each continue on the shared account in a new container that resumes the session, an Enhanced run with nowhere to go parks with the no-fallback reason, and TestRestartSettlesAnInterruptedAccountMove restarts mid-move. The signals are pinned against vendor-shaped fixtures and their near misses: TestUsageLimitFromPromptError (internal/acphost/testdata: Claude, Codex and opencode prompt errors), TestClaudeUsageLimit (internal/adapter/testdata/claude_usage_limit.jsonl and claude_rate_limited.jsonl) and TestClaudeStopFailureUsageLimit (internal/agentstatus). TestCostStaysWithTheAccountThatSpentIt (internal/cost) keeps each segment's cost on its own account, through a run deletion. make test TEST_PKGS=./internal/store covers concurrent database opens against real SQLite files. The migration contention regressions keep a competing writer active while checking startup resumption from committed progress, the newer-schema error and its version bounds, foreign-key restoration, and refusal to trust an uncommitted version. TestConcurrentOpen still races eight opens on a fresh database. internal/push proves Web Push against a fake push service rather than a real one: TestSealMatchesRFC8291Example reproduces the RFC's own message byte for byte, TestSendIsAStandardWebPushRequest verifies the VAPID token against the key it names and decrypts the body with the subscription's keys, and service_test.go drives a real store and bus through what is announced, held, not repeated and forgotten.

• Integration/E2E tests are behind the integration build tag and run with make test-integration (real Docker, real git), which covers only the packages carrying integration-tagged tests. INTEGRATION_PKGS narrows that to one package and INTEGRATION_SKIP leaves some out. INTEGRATION_RUN and INTEGRATION_SKIP_PATTERN, when set, append -run and -skip. CI runs on GitHub-hosted runners, with GOFLAGS=-v so each test's duration is in the job log. The integration matrix in .github/workflows/ci.yml has five shards that need no browser image: server-chaos (INTEGRATION_RUN=^TestIntegrationChaos), server-mission-coordination (INTEGRATION_RUN='^TestIntegration(Mission|Coordination)'), server-heavy (INTEGRATION_RUN='^TestIntegration(EndToEnd|MultiMember|ServerUpdate)'), scheduler (INTEGRATION_PKGS=./internal/scheduler), and rest. The three named server shards use INTEGRATION_PKGS=./internal/server. rest dynamically discovers integration packages, excluding ./internal/harness, ./internal/scheduler, and ./internal/server; it skips ^TestDockerBrowser, whose coverage belongs to the native amd64/arm64 browser jobs. The separate integration-server-rest job builds images/browser/Dockerfile itself, reading the browser-amd64-main build cache that the browser job writes on pushes to main, and runs ./internal/server with INTEGRATION_SKIP_PATTERN='^TestIntegration(Chaos|Coordination|Mission|EndToEnd|MultiMember|ServerUpdate)'. This catch-all retains tests that do not start with TestIntegration. No integration job waits for another job's image, so all of them start when changes finishes. Both integration job families use sudo env, preserving PATH, HOME, GOFLAGS, AETHER_BROWSER_TEST_IMAGE, and AETHER_BROWSER_IMAGE. The separate unit suites remain unprivileged: the root-only TestApplyRunOwnershipHardlinkSafe and both privilege branches of TestRecoveredTerminalUsesCapturedUserAndHomeForImages must stay covered. The smoke job runs internal/harness on the images it builds natively on amd64 and arm64, alongside scripts/standard-image-smoke.sh toolchain and native Git checks. A docs-only pull request (only docs/** and root *.md) runs audit and skips every other job, including these, the dashboard jobs, and release-build. A markdown file anywhere else, including a dashboard end-to-end fixture, does not. The changes job runs scripts/ci-classify-changes.sh from the pull request's base revision, so a change to that script cannot make the decision itself, and a rename is classified by both its old path and its new one. The edge-image job calls .github/workflows/edge-image.yml, which builds images/edge/Dockerfile on an amd64 and an arm64 runner and runs scripts/edge-image-smoke.sh on each (The edge image); it pushes nothing and uses no build cache. A pull request runs it only when scripts/ci-classify-edge.sh, also from the base revision, says a changed path can affect the image; a push to main, the merge queue and a failed classification always run it. A rerun classifies the same files, so to force it on a branch the classifier skips, start the workflow by hand, which needs write access to the repository:

  CODE

  gh workflow run edge-image.yml --ref <branch>
  gh workflow run edge-image.yml --ref main -f ref=refs/pull/<n>/head   # a pull request from a fork

  These jobs are the merge gate the E2E suite owns. The runtime capacity scenario starts its own Docker and containerd daemons with private sockets, storage roots, and network/mount namespaces; it never reconfigures the host's daemon or network. Go's child Unshareflags setup makes mount propagation recursively private before exec. The fixture verifies that each daemon's network and mount namespace identities differ from the test process, and that an existing host docker0 bridge retains its identity, MAC, MTU, and administrative state through startup and teardown. It needs root with permission to unshare these namespaces, plus dockerd and containerd on PATH. Docker's info API must expose Containerd.Address and namespaces (API 1.46+, including Docker 28's API 1.48); missing metadata fails the proof, rather than skipping it or guessing an address. The fixture uses the documented --containerd-plugins-namespace CLI flag, because Docker 28/29's singular JSON tag does not match their flag validation. Containerd's introspection API must export the actual content and overlayfs roots, including the fixture's custom root_path override. Both daemon readiness waits observe process exit immediately and retain their owned logs on failure; teardown uses bounded TERM/KILL waits. The other resource scenarios observe actual cgroup limits, PID exhaustion and OOM containment, not agent-session isolation:

  CODE

  sudo env "PATH=$PATH" "HOME=$HOME" go test -race -tags=integration \
    ./internal/runtime \
    -run '^TestDocker(CapacityContainerdIsolated|ResourceLimits|PIDLimitEnforced|OOMContained)$' -v

  Classic-driver capacity unit tests also verify the separate layer-store filesystem: Moby stores overlay2 layers under data-root/overlay2 and VFS layers under data-root/vfs/dir. Both driver homes are created during initialization, but VFS creates dir only with the first layer; before that, capacity uses the verified vfs parent. Missing driver homes, broken symlinks, and non-directory layer stores fail closed. The cross-filesystem regression resolves an owned symlink to an existing /dev/shm filesystem, without creating mounts or changing shared paths, and requires aggregate storage capacity to remain unknown when layer and data roots differ.

• Dashboard component tests live beside their components in web/src/ and run with bun run test from web/ (vitest in jsdom). CI runs them in the dashboard job. jsdom has no layout, so web/src/test/setup.ts answers every media query with false and a component renders its widest branch; atViewport (web/src/test/viewport.ts) puts one test on one screen instead - width and pointer queries answer for it, window.innerWidth/innerHeight report it, and the returned resize fires change where an answer moved and resize on the window. It decides which branch renders and nothing more: real layout belongs to the browser suite below. setup.ts also empties the rules of every stylesheet whose <style> element has been detached. jsdom unregisters a sheet only while that element is still connected, and xterm discards its three style elements with the terminal around them, so each terminal a file mounts strands three sheets of roughly 800 rules. The entries themselves stay in document.styleSheets and their count still grows; emptying their rules is what keeps the cost flat, because getComputedStyle - which every testing-library visibility query runs - re-cascades only the rules that are still there.

• Dashboard end-to-end tests live in web/e2e/ and run with make test-e2e: a real browser driving the static Next export embedded by the shipped binary, through a real aether gui gateway and a real aether-server. They own the paths a person walks in the dashboard, which no Go test and no jsdom test reaches. CI runs four isolated dashboard-e2e runners, each with its own Docker daemon and Playwright workers: 1. Each runner builds the browser and standard images itself, reading the browser-amd64-main and standard-amd64-main build caches that the browser and smoke jobs write on pushes to main, so it starts without waiting for those jobs. A change to an image's Dockerfile or its inputs misses that cache from the changed step on, in its pull request, its merge queue entry and the push to main that lands it: those runs read the cache before browser and smoke write the new layers. In each of them every shard, and integration-server-rest, rebuilds those layers itself, network downloads included. Each runner then runs make test-e2e with E2E_ARGS='--shard=1/4' (or 2/4, 3/4, 4/4). Local make test-e2e remains unsharded and still builds the embedded dashboard and binaries first. The four shards partition the desktop and mobile cases once each. fullyParallel: true makes --shard split by test rather than by file; with one worker, tests still run one at a time, and a file split across shards runs its beforeAll hooks in each. The chromium project excludes **/*.mobile.spec.ts; only the mobile project owns those specs. The opt-in real-GitHub case keeps its existing credential gate; listing or sharding it does not prove it ran. Each shard retains playwright-report-<shard> (from web/playwright-report/) and playwright-results-<shard> (from web/test-results/) for seven days on success or failure, including phone screenshots and failure evidence. Only shard 1 may save the shared Playwright browser cache, on a non-cancelled push to main; a cache miss never skips browser installation or tests. Remote-development scenarios live in web/e2e/development-browser/ (shared real companion, login, live app update, popups, takeover and phone viewport input), web/e2e/development-terminal/ (agent-created shared TUI, protocol replies, control, geometry and process lifetime), and web/e2e/remote-development-git/ (remote import, native selected-path commit and push through the Changes view's Publish dialog, plus opt-in actual GitHub PR publication). These use deterministic harness fixtures, not authenticated vendor-agent loops.

CI's native windows job builds, vets, and tests the full Windows client package closure (./cmd/aether and its repository dependencies). It owns the selected internal/localops regressions rather than repeating them in the installer lanes. TestInstallDesktopWindowsPreservesCLI requires install and reinstall to preserve the CLI and unrelated files in the documented CLI directory and detect only the desktop directory as an installed app. TestDesktopLayoutWindowsUsesProgramsKnownFolder covers standard and redirected Programs locations independently of missing, relative, or ordinary APPDATA. TestInstallDesktopWindowsProgramsLookupFailurePreservesInstall checks that a failed lookup leaves an existing installation intact. Unit installation tests inject a temporary Programs destination and never overwrite the developer's real Start entry. TestShellLinkLaunchesNativeConsumer separately exercises Windows' real shortcut launcher with a target path containing spaces, an ampersand, and non-ASCII characters, and checks the launched process's working directory. These tests do not establish shell catalogue discovery or keyboard Search behavior.

CI calls the reusable, workflow_call-only Windows install workflow under the same code-change condition as the native windows job. Installer results are therefore part of the main CI run required by the release gate, not a separate workflow whose result can be omitted from that gate. Docs-only changes skip both jobs; code changes, including Go/Bun cache action changes, include both. The installer workflow runs scripts/install-test.ps1 under Windows PowerShell 5.1 and PowerShell 7. Those scenarios cover checksum rejection before replacement, upgrade and locked-file boundaries, PATH preservation, CLI-only installation, unsupported releases, and desktop-build failures. They use temporary files and restore the user's environment after running.

The same workflow builds the real CLI with Windows release metadata and the embedded dashboard while scripts/windows-defender.ps1 -Mode StartUpdate downloads Defender definitions in the background. It then runs -Mode Prepare, which removes the hosted runner's inherited exclusions, enables Defender realtime, script, archive, and first-seen cloud scanning, and waits for that update before running scripts/install-smoke.ps1. A local release mirror labels the checkout build v0.5.1-alpha.4 and serves its exact bytes and checksum; it does not download or claim to test the historical published release with that tag. The smoke refuses to run unless both GITHUB_ACTIONS=true and RUNNER_ENVIRONMENT=github-hosted, before changing files or registry state. It installs and reinstalls into an isolated app/config tree and verifies the unchanged CLI. It retains the real hosted user's USERPROFILE, HOME, and APPDATA so child-process Known Folder expansion matches the shell. LOCALAPPDATA is redirected only through installation and the private-Node check, then restored before creating Shell.Application, querying AppsFolder, or activating either launch. The captured installed paths, Aether configuration, npm/Electron build caches, and explicit Electron user-data directory remain isolated; restoring the environment does not move the installed executables. Unlike the unit test's injected destination, it resolves the real current-user Programs Known Folder through Environment.SpecialFolder.Programs, backs up any existing Aether.lnk, and restores it in finally before deleting the temporary app. It never redirects permanent shell-folder registry settings.

For each install, the smoke inspects the real shortcut's executable target and working directory. It then waits up to 90 seconds for Shell.Application's shell:AppsFolder catalogue to enumerate Aether with the installed executable as its link target, rather than accepting a filename or display-name match. Discovery and activation timeout diagnostics include the catalogue item count, the total number of Aether candidates, and at most ten candidate names, paths, AppUserModelIDs, and targets. They also report the real LOCALAPPDATA, Programs and fixture paths, the caller's session and interactive status, and at most eight Explorer process IDs/session IDs with same-session and total Explorer counts. There are then two launches, with the caller's pre-install PATH and no AETHER_BIN override:

1. The smoke invokes InvokeVerb('open') on the exact discovered AppsFolder item. Because this verb has no argument parameter, the fixture shortcut temporarily carries only an added --user-data-dir isolation argument; its installed target and working directory are rechecked. The smoke waits for that exact installed executable's main window and CLI child, closes the window normally, and requires both processes to exit within bounded waits before restoring the shortcut arguments.
2. It launches the matching Programs shortcut with explicit remote-debugging and isolated user-data arguments. Playwright checks that window's onboarding screen, saves a screenshot, and closes it; both desktop and CLI child must exit.

The protocol registration is backed up before either launch and restored in finally, which also stops any surviving fixture desktop/CLI processes. Explorer-mediated catalogue activation may inherit Explorer's environment rather than the caller's config overrides: the real profile is deliberately a disposable hosted profile, never a developer's profile. The explicit Electron data directory applies to both launches. This proves native shell catalogue discovery, catalogue activation, and shortcut onboarding, not keyboard-driven Start Search indexing or ranking. Shell COM works in both supported PowerShell lanes without relying on a PowerShell-5-only module. Windows PowerShell 5.1 uses system Node; PowerShell 7 (pwsh) hides system Node to exercise the verified private download. The seven-day screenshot artifacts are windows-desktop-powershell-system and windows-desktop-pwsh-downloaded, uploaded even after failure when a screenshot exists. Defender scans the download and install tree without exclusions or disabled remediation. The gate rejects new detections even when Defender has already remediated them, requires automatic safe sample submission (SubmitSamplesConsent=1) before and after installation, and checks that installation changed neither protection settings nor exclusions. This is a detection gate, not a guarantee that an unsigned release will never receive a false positive on another machine.

Workflow and release-build gates

Before installing Playwright's system dependencies or desktop validation tools, CI sets /etc/apt/apt-mirrors.txt to https://archive.ubuntu.com/ubuntu/. This bypasses the hosted runner's azure.archive.ubuntu.com mirror, which can stall apt-get update until the job times out. Both jobs still install their required packages.

make lint-workflows runs pinned actionlint v1.7.12 and requires ShellCheck on PATH so embedded shell commands are checked locally as well as in CI. On Linux, install it with sudo apt-get install shellcheck; GitHub's Ubuntu runners already provide it. CI invokes this target in the lint job. make test-scripts includes sh scripts/release-ci-check-test.sh, which exercises the real release checker's run selection and rejection behavior with only the GitHub API boundary replaced. The checker requires the latest matching run/current attempt of this repository's .github/workflows/ci.yml to be a completed, successful push on main for the exact full release commit SHA. Missing, malformed, or failed API responses and a newer pending or failed run reject publication; an older success, PR run, or merge-group run cannot substitute. See release instructions for the publishing gate.

CI's release-build job runs make release-binaries, which builds all ten Go assets in parallel: server and edge for Linux, and the CLI for Linux, Darwin, and Windows, each for amd64 and arm64. It uploads the two Windows clients as binaries-go-windows and the other eight as binaries-go. The independent android job validates the Gradle wrapper and runs make android, uploading aether-android-unsigned.apk and aether-android-unsigned.aab in binaries-android. All three artifacts have seven-day retention and missing outputs fail upload. Locally, make release-binaries builds the same ten assets without Android or Docker; make release still adds Android. Go builds still need the dashboard's Bun and Node toolchain.

windows-defender depends on release-build, not Android. It downloads only the current run's binaries-go-windows artifact and requires both aether-windows-amd64.exe and aether-windows-arm64.exe. CI and release publication share scripts/windows-defender.ps1: prepare the hosted runner before downloading artifacts, remove inherited antivirus exclusions, enable and verify effective realtime/cloud protection, automatic safe sample submission, and all required scanning flags, and update definitions. Fresh runners update the Defender platform shortly after boot, which restarts WinDefend, so Prepare waits up to five minutes for the service and for real-time protection to report enabled. It logs and retries every error from starting the service or reading its status until that deadline, then fails with the last status and error. Reading the detection baseline, changing preferences, updating signatures, and verifying protection retry only 0x800106ba (service unavailable) and 0x80070652 (another update installing); any other error fails immediately. The helper retains each binary's resolved absolute path for hashing and the native scan, scans with remediation enabled, records each binary's SHA-256 and Defender status/definitions in the logs and job summary, and rejects missing or quarantined files, changed hashes, scan failures, and new detections even when already remediated. Scan or pre-scan hash failures include the Defender log path and up to its last 200 lines for diagnosis. The release workflow scans the final Windows artifacts from its own release-binaries build without rebuilding them; publication depends on that exact-byte scan succeeding. A scan of CI's earlier build cannot stand in for this release scan. Windows executables remain unsigned. These static scans supplement, rather than replace, the runtime installer Defender scenarios above. Android and every Go target remain required CI coverage.

Go caches separate module downloads from compiled objects. Both are scoped to host OS/architecture, runner environment, resolved toolchain, and module digest; compiler caches also isolate the lane and commit, with fallback only within the same lane. Only pushes to main write: build-and-test, windows, and arm64 smoke own their respective host's module/native compiler archives; release-build owns the release compiler archive, which the release workflow's per-platform lanes restore. PRs and releases restore only. The installer uses a separate windows-install compiler lane; only its PowerShell/system-Node main lane saves that cache, leaving Windows module writes to CI's windows job. It saves right after the build, before arming Defender, whose realtime scan made archiving take eight minutes. The installer restores no Bun cache: on Windows, restoring the 115 MB archive took 50 seconds at the median and 90 at p90, longer than the 39-second dashboard build it served. Before compressing an archive, each writer checks whether its exact key already exists and skips the save on a hit. Cache reuse does not replace any build or validation gate.

Running the Docker suites inside a run

On a server host with Sysbox (environments.md), a run or Environment terminal has its own dockerd, so the Docker-backed suites run from inside it. Bind-mounted temp paths resolve because the daemon and the test process share one filesystem, which is not true when the socket belongs to the host. Check first with docker info.

make test-integration                    # real Docker, real git; Go and git are in the image
make android                             # needs only the docker CLI and network
curl -fsSL https://bun.sh/install | bash # make test-e2e also needs Bun on PATH
(cd web && bun install --frozen-lockfile && bunx playwright install --with-deps chromium)
make test-e2e

To prove the path on a Sysbox host, install Sysbox with sudo sh scripts/install-sysbox.sh, build the standard image, then run the Go test against it and the smoke script:

docker build -f images/standard/Dockerfile -t aether-standard:ci .
sudo env "PATH=$PATH" "HOME=$HOME" \
  AETHER_NESTED_DOCKER_TEST_IMAGE=aether-standard:ci \
  make test-integration INTEGRATION_PKGS=./internal/runtime \
    INTEGRATION_RUN='^TestDockerNestedDocker$'
sudo sh scripts/nested-docker-smoke.sh aether-standard:ci

TestDockerNestedDocker creates a container with the runtime the server resolves, starts dockerd in it and runs a nested container. It skips unless AETHER_NESTED_DOCKER_TEST_IMAGE names an image that contains dockerd. scripts/nested-docker-smoke.sh starts a Sysbox container with the checkout mounted at /src, starts dockerd in it, and runs TestDockerResourceLimits, TestDockerPIDLimitEnforced and TestDockerOOMContained against the nested daemon. That proves cgroup limits and Aether's capacity checks hold on a daemon that is not the host's. It pulls busybox and Python images, so it needs network access.

Headless browser and remote-development acceptance

These are commands and acceptance requirements, not a record of a completed smoke run. Use a stock headless Ubuntu Docker host with the source-build toolchain from install.md. No display session, X11/Wayland, Xvfb, host Chromium, or host browser libraries are needed by the companion. Build and exercise the exact native image:

docker info
docker build -f images/standard/Dockerfile -t aether-standard:ci .
make browser-image
make browser-smoke

# Match the installed server's authority over its private UID-1000 bind.
sudo env "PATH=$PATH" "HOME=$HOME" \
  AETHER_BROWSER_TEST_IMAGE=aether/browser:test \
  go test -race -timeout=10m -tags=integration ./internal/runtime \
    -run '^TestDockerBrowser' -v

# Load/build that image in this daemon before the complete integration gate.
AETHER_BROWSER_TEST_IMAGE=aether/browser:test \
AETHER_BROWSER_IMAGE=aether/browser:test make test-integration

# Install the dashboard driver's Chromium once; it is separate from the companion.
(cd web && bunx playwright install chromium)

# Real dashboard interaction through the built gateway/server, as root like the
# installed service. Without sudo the suite runs the server as your uid; a
# passing test's teardown chowns what root containers left back to you with
# the scenario's standard image before deleting its scratch directory. A failed
# scenario keeps /tmp/aether-e2e-*, root-owned files included: remove it with
# sudo rm -rf.
sudo env "PATH=$PATH" "HOME=$HOME" \
  "PLAYWRIGHT_BROWSERS_PATH=$HOME/.cache/ms-playwright" \
  AETHER_E2E_STANDARD_IMAGE=aether-standard:ci \
  AETHER_BROWSER_TEST_IMAGE=aether/browser:test \
  AETHER_BROWSER_IMAGE=aether/browser:test make test-e2e

make browser-smoke BROWSER_IMAGE=<reference> exercises another exact image. The image contains the scripts, Playwright, Chromium, OS dependencies and fonts; no source or member-home bind participates. The smoke reads Linux /proc/<pid>/status to verify nested renderer PID namespaces, no-new-privileges and Chromium seccomp filters beyond Docker's filters. It interacts with a loopback app, captures a browser PNG and a bounded JPEG frame, resizes while an observer remains attached, and checks that a new observer receives the new-size frame. It also exercises a popup and console report, renders a terminal PNG with screen metadata, and resets the browser context. The companion's additional behavior checks force a resize frame to arrive before the resize operation returns and cover unchanged-size viewport requests. These checks run in the same image. A Docker spec or requested launch flag alone is not proof of a working Chromium sandbox.

The runtime integration command separately exercises real namespace sharing, the private control mount, immutable image identity, lifecycle recovery and the absence of a CDP TCP listener. Run it with the installed server's root authority; do not make its control directory world-writable to get a pass. Sandbox failures must fail the gate. Inspect the host's kernel/Docker/custom AppArmor diagnostic; do not retry unconfined, with --no-sandbox, or after a global security-policy relaxation.

CI's browser jobs use native Ubuntu amd64 and arm64 runners, building locally on pull requests without registry writes. Releases smoke the exact architecture images before publishing and require anonymous pulls of the versioned multiarchitecture manifest. An administrator must make the new GHCR package public before that gate can succeed; neither this guide nor a workflow definition proves publication or a successful run.

The E2E native Git scenarios require the real aether-standard:ci image built from images/standard/Dockerfile (or the exact image selected by AETHER_E2E_STANDARD_IMAGE), with native Git and gh; a fake executable is not a substitute. Browser scenarios require the real companion image in the same Docker daemon and the loopback app image node:22.14.0-alpine3.21 available to pull or already loaded. The Playwright dashboard driver has its own Chromium and host dependencies; the companion's lack of host-browser prerequisites does not waive the driver's setup. Build/load prerequisites and run commands are not evidence that these suites have passed.

web/e2e/remote-development-git/github.spec.ts is a separately gated, externally visible smoke. It runs only with AETHER_E2E_GITHUB_PUBLISH=1, AETHER_E2E_GITHUB_TOKEN, AETHER_E2E_GITHUB_BASE_REPOSITORY (a public owner/name) and AETHER_E2E_GITHUB_HEAD_REPOSITORY (a writable fork owner/name); AETHER_E2E_GITHUB_BASE_BRANCH defaults to main. Supply the token through the test environment, never documentation or logs. Its authority must allow pushing the fork and creating, commenting on and closing upstream PRs. The scenario creates and closes a real PR and removes its temporary head branch. An absent credential, disabled opt-in or skipped smoke is an explicit GitHub coverage gap; native/local publication tests do not replace it.

Native Git boundary for commit coverage. Managed selected-path commits use native prepared git update-ref --stdin transactions inside the run. The standard image uses Ubuntu 24.04's distribution Git, not a special source build. Stock Git 2.43 supports the required native transaction. No custom Git build or new Ubuntu host Git upgrade is required.

go test ./internal/runrepo

This suite exercises native Git and native gh against an isolated TLS API fixture. It does not replace Docker, authenticated GitHub, or real-agent acceptance.

Verify rejection of a changed symbolic HEAD and expected branch OID, including a branch switch to another branch at the same OID. Exercise native lock contention and abort behavior, selected-path exactness, unrelated staging preservation, native identity/signing/coauthors, and the distinct committed=true, index_updated=false outcome. Custom environments lacking transaction support must fail closed and retain native diagnostics; see install.md. Do not label selected-path commits commit-hook-aware: their result is hooks_run=false; use native git commit in the run terminal when commit hooks are required. Native reference-transaction hooks must remain active, including a preparation veto, and hook output must not be mistaken for protocol acknowledgements.

End-to-end acceptance matrix

Exercise the following against both a local aether gui gateway and the server-hosted HTTPS dashboard at desktop and phone viewport widths. Record separately whether real tailnet identity, mobile operating-system keyboard, and WebView behavior were exercised; viewport emulation does not prove those.

• Start a real app in a named development terminal in the live run. Observe initial output, reconnect/replay, screen and geometry changes, normal and alternate-screen rendering, terminal capture, exit and explicit restart. Confirm a phone follows the shared terminal without silently resizing it.
• Open the app's run-local URL in the companion, use a real DOM snapshot and action, navigate and switch pages, inspect console/request failures, view streamed frames, change viewport, and capture evidence. Join a static page after another observer and confirm the latest complete frame appears. After navigation, reset, closure, or restart, old page/node/viewport references must fail rather than act on another target.
• Let a run agent own an app surface, explicitly take it over as a human, confirm stale agent input is rejected, then release/reacquire it. A second terminal and the browser must remain independently controlled; the primary agent's swarm hold must remain intact. Hold browser input across release, revocation and controller disconnect; confirm server-side cleanup and refusal of replacement control if cleanup fails. Observer disconnect must not clear a live controller's input.
• Confirm read/capture/stream and write authorization on both gateways, including a member without the steer permission, backing-account revocation, lifecycle stop/pause, stale control generations, and reconnect after server restart. Browser loss must not silently replace an authenticated session.
• Exercise real Git status/diff and an explicit selected-path commit in the run environment; verify signing and branch identity, concurrent branch rejection, commit-hook disclosure, native reference-transaction hook behavior, and fail-closed transaction diagnostics. Confirm no browser profile or capture enters the source tree or commit.
• Inspect deliberately selected evidence, its attribution and retention. Retain reviewed captures and bounded notes before headless report/cleanup, read back the packet and download its exact bytes using evidence_packet_id. Confirm workspace View can read retained evidence while private transient access still requires steer/account-use. Exercise queued revocation and busy publication admission with explicit retry, retained-plus-staged quotas, transient deletion, and the separate capture-time and later packet Git boundaries. Keep credential entry out of recordings and confirm nothing uploads images to a public PR by itself.
• Cancel or revoke idle browser streams and capture downloads through both transports, including a stalled SSH peer. Source cleanup must not wait for blocked status/close writes; responsive connections must preserve sibling channels. Interrupted downloads must not look like complete captures.
• Complete two genuine authenticated vendor-agent loops through the shared development tools, with observed terminal/app changes and reviewed evidence. Record which vendors, image identities, gateways and phone were exercised, and any missing prerequisites or unexercised rows.

Public CI has no authenticated real-agent credentials. Its deterministic fake agents and scripted claude, pi, or omp fixtures prove their stated broker/transport paths, not two genuine vendor loops. No-login vendor smokes prove argument acceptance and login failure, not authenticated work. A skipped test, unavailable Docker daemon, fake runtime, or absent phone is an explicit coverage gap, never a passing real-agent or phone acceptance.

Local configuration in tests

Tests that read or write the linked-server config through cli.Load, cli.Save, or gateway handlers must call internal/testhome.Isolate(t). It sets AETHER_CONFIG_DIR, the platform home/config variables, and clears SSH_AUTH_SOCK. Setting only XDG_CONFIG_HOME and AppData leaves the real macOS config at ~/Library/Application Support/aether/config.json exposed. The gateway regression in internal/localgw/config_isolation_test.go runs config refresh and repository linking against a temporary user config and checks that its contents and modification time stay unchanged, including when the test process inherits AETHER_CONFIG_DIR.

The E2E scenario suite

internal/server's *_integration_test.go files are the owned end-to-end suite: every scenario drives the fully wired server (server.New) over real SSH, real git transport, and - when the daemon is reachable - real Docker containers. pickRuntime falls back to the in-process e2eRuntime (e2eruntime_test.go) on hosts without Docker; the two host-half coordination scenarios force it because their agents must reach container surfaces from the test process, and the container coordination scenario skips without a daemon rather than falling back. Scenarios:

The chaos scenarios

chaos_reboot_integration_test.go is the one place the suite runs aether-server as a child process. An in-process server cannot be SIGKILLed, and the whole point of that row is that nothing on the shutdown path runs: the next boot only sees what SQLite and git had already made durable. The child binary is built once per test binary, the store is seeded before startup, and it binds a reserved loopback port so a restart can claim the same address. It always builds its own Docker runtime, so these scenarios skip without a reachable daemon rather than falling back, and they address containers by the name the runtime derives from the run ID.

Disk pressure is driven through configuration, never by filling the host disk: --min-free-disk above what the machine has free makes the real statfs path refuse for real, and --checkout-ttl turned down makes the real GC sweep on the next boot. No injected filesystem, no fake statfs.

The SSH-drop boundary is fuzzed where it lives, in internal/ptyhost (drop_fuzz_test.go): the connection is cut at every byte offset and the agent's stdin must always be an exact prefix of what the transport delivered, which catches a byte from past the cut as well as any reorder or duplication. Cutting the stream cannot reach the other half of that row - the read loop stops at the first error and never asks a dead connection for more - so the case where the attach unwinds first and the socket coughs up a straggler afterwards has its own scenario in the same file.

The deterministic fake agent is the scheduler's fake harness: its argv comes from AETHER_FAKE_AGENT at launch (typically sh /workspace/agent.sh {task}, with the script committed to the seed repo and dispatching on the task). On the fallback runtime the same behaviours are registered per task key via e2eRuntime.script.

Three server.Config fields exist for this suite: WhoIs overrides tailnet identity resolution so join and fallback scenarios need no real tailnet, Harnesses overrides registry argv templates so a registered harness (with its real profile root and credential mounts) can run a scripted agent - the first two double as deployment wiring - and ServerBinary names the executable staged for the CLI and optional MCP bridge.

The container coordination scenario

coordination_container_integration_test.go proves the half of docs/mcp-bridge.md that an assertion on a container spec cannot: that the mounts a run is given are real, and that the agent holding them can use them. Two things make it possible.

The staged bridge has to be a binary that has the mcp subcommand, which under go test /proc/self/exe is not. So the scenario points ServerBinary at an aether-server it builds - the same one the chaos scenarios run as a child process.

The scenario uses the shipped claude profile with a non-root fixture agent from internal/server/testdata/coordagent. It explicitly invokes the optional bridge from the canonical executable mount; no automatic MCP registration is required. The fixture reports directory modes, read-only mounts from the kernel's mount table, EROFS on attempted writes, and tool results over a real attach. The daemon's mount view is checked alongside those observations.

TestIntegrationMissionCompositionInDocker composes swarm dispatch, proactive coordination, accepted retained submissions, combined verification, a delivery request approved without a human decision, and exact delivery. It also checks failed verification, stale-target rejection, and integrator replacement. Its claude, pi, and omp executables are scripted fixtures, not genuine vendor-agent runs. web/e2e/mission-candidate-review.spec.ts drives launch, progress, and worker control in a real browser. After a successful worker report, it checks that the same Docker container remains paused. The integrator then prepares a candidate through its coordination CLI, and the already open swarm page must list the worker's report under Agent messages and show the candidate read-only under Integration, without a reload. It attaches a successful screenshot for visual inspection.

web/e2e/run-room.spec.ts sends structured request snapshots through the staged reporter in a real container. Both members' browsers must add and clear the request; while it is open the owner's run header reads Question: answer in the terminal and the other member's reads Waiting for the owner. The callback payloads are scripted fixtures, not evidence of a live vendor agent emitting them.

The container user is the test process's own uid:gid unless that is root: the scheduler chowns the run checkout and the member home to the container user before creating the container, and an unprivileged test process can only chown to itself.

The real-agent smoke tests

internal/harness/smoke_integration_test.go launches the vendors' actual CLIs and checks that the argv Aether ships is still the argv they accept. Nothing else catches a vendor renaming a flag or refusing a combination it used to allow: a change like that breaks every run of that agent and no amount of internal testing sees it coming.

TestSmokeHeadlessNoLogin is the one that runs in CI. It launches each headless template with no credentials at all and requires the run to fail for want of a login, never for want of a parseable command line - reaching the provider is proof the CLI accepted the flags. opencode needs no login, so its proof is the > build session header it prints before its first call to its free hosted model. The test destroys each container as soon as the proof appears, so no result depends on a model answering. On failure it prints the output captured so far. The smoke job builds images/standard/Dockerfile, then images/smoke/Dockerfile on top of it to add the agent CLIs at whatever version their vendors ship that day, and points the gate variables at the result.

To run it locally, build the same image and name it:

docker build -f images/standard/Dockerfile -t aether-standard:local .
docker build -f images/smoke/Dockerfile --build-arg BASE=aether-standard:local \
  -t aether-smoke:local .
AETHER_SMOKE_IMAGE_CLAUDE_NOLOGIN=aether-smoke:local \
AETHER_SMOKE_IMAGE_CODEX_NOLOGIN=aether-smoke:local \
AETHER_SMOKE_IMAGE_OPENCODE_NOLOGIN=aether-smoke:local \
  go test -tags integration -run TestSmokeHeadlessNoLogin ./internal/harness/

The _NOLOGIN images must carry no credentials. The other smoke tests - TestSmokeClaude, TestSmokeOpencode, TestSmokeCodexFlags - drive the agent far enough to produce output, so they need an image that does carry that agent's login state, named by AETHER_SMOKE_IMAGE_<NAME>. CI has no such credentials, so those stay a manual check. Every one of them skips when its variable is unset, and the no-login test skips as a whole rather than reporting a pass with nothing run.

pi and omp have no template smoke in that image. The native mailbox lifecycle checks below are separate from template acceptance and authenticated TUI smoke.

The status extension those two load, internal/agentstatus/status.ts, is the one shipped file the Go build never executes. internal/agentstatus runs it under bun against a recording stand-in for the server binary, so a turn's reports are proven to come out in order and to end exactly once. Those tests skip where bun is not installed.

TestACPAdapterInstall installs the pinned Claude Code and Codex enhanced-mode adapters into a home mounted at /home/aether in the standard image, as uid 1000, and completes the ACP initialize handshake with each, logging the cold start. It needs the npm registry and skips without it:

AETHER_ACP_IMAGE=aether-standard:local \
  go test -tags integration -run TestACPAdapterInstall -v ./internal/harness/

With ACP_LIVE=1 it also opens a session with each adapter through the session host (initialize then session/new, no prompt), so no login is needed; a logged-out Codex skips at session/new.

Enhanced runs are proven without a real agent: internal/acphost/acpmock replays conversations recorded from the real adapters and answers prompts by their text, so tests can drive a permission request (ask permission), a form question (ask form), a turn that runs until cancelled (wait), Claude's usage-limit prompt error (hit the usage limit) or a paced turn with a plan, tool calls, command output and a diff (demo; demo edit also writes that edit to the session's working directory). The scheduler's unit tests run it behind the fake runtime; TestIntegrationEnhancedRunDocker (scheduler) and TestIntegrationEnhancedRunGateway (server) build it as a static binary and run it as the ACP server of a real container, the second through /ws/acp/<run_id> on the server gateway:

make test-integration INTEGRATION_PKGS='./internal/scheduler ./internal/server' \
  INTEGRATION_RUN='TestIntegrationEnhancedRun'

The gateway scenario reports success and failure between turns, then sends follow-ups through the same ACP stream and checks that the agent session ID is unchanged. TestIntegrationCoordinationCLIFromShellHarnesses reports from inside real Standard containers, sends follow-ups through their existing raw terminal attachments, reports again, and checks that only explicit Close starts the seven-day retention grace.

web/e2e/run-session.spec.ts drives the dashboard's Session view against the same agent: the e2e fixture installACPMock builds it (go build, so the e2e host needs Go) into the member's ~/.local/bin and registers it as the member agent mock. The spec launches an Enhanced run, watches the stream, answers a permission with the digit key, expands a work entry's diff, opens the edited file from the changes bar, checks the bar again after a reload and interrupts a turn. Its two-member scenario checks shared notes, moderated instructions, occupied-click refusal, early hold cancellation, holder denial, automatic transfer after the five-second hold and seven-second review, explicit release without reacquisition, controls across view switches and a narrow-screen layout:

make build && (cd web && bunx playwright test run-session.spec.ts)

TestSupervisorSwapsItsChild waits for the fallback login shell to execute a readiness command before requesting another swap. The harness exit banner is printed before that shell starts. Signal fixtures register a buffered SIGTERM or SIGHUP channel before announcing readiness; a login shell under test execs one instead of trapping the signal, because a shell trap stays pending when the signal lands just before read blocks. After the forced swap, the test executes a command in the replacement login shell.

Mode switching is proven the same way. TestIntegrationModeSwitchDocker switches a run in a real container both ways, with acpmock as omp's ACP server and a script as its terminal. TestIntegrationSupervisorSwapsInContainer swaps the run supervisor's child under busybox's sh and, with AETHER_ACP_IMAGE set, under the standard image's dash. TestLiveSwitch is the check behind an agent's switchable: with ACP_LIVE=1 it runs the agents and adapters on the host's PATH with their own logins, starts a session over ACP and asks about it from the terminal, then the reverse. ACP_LIVE_OMP_MODEL picks omp's model when its default cannot answer:

ACP_LIVE=1 go test -tags integration -run TestLiveSwitch -v ./internal/harness/

TestLiveEnhancedRunsWithoutPermissionPrompts opens a Claude Code and an omp session the way an Enhanced run does and has each run echo ok > ran.txt in its shell; a session/request_permission or a missing file fails it:

ACP_LIVE=1 go test -tags integration -run TestLiveEnhancedRunsWithoutPermissionPrompts -v ./internal/harness/

dev.port.list reads a run's listening ports with a shell script inside its container. TestListeningScriptFindsThisProcess runs that script against the test machine's own /proc, TestParseListeningPortsFromACapturedRun parses its output captured in a run with dual-stack, IPv4-mapped, loopback and forked listeners, and TestIntegrationPortListDocker launches a run that serves one port on every interface and one on loopback under busybox and reads both back.

Native mailbox lifecycle and idle-wake smoke

Run the shipped adapters against deterministic SDK-shaped lifecycle fixtures with Node 22.13 or newer:

make test-native-hooks
node --experimental-vm-modules --test --test-name-pattern='omp:' \
  internal/coordhooks/native_pi_omp_lifecycle_test.mjs

The OMP fixture separates extension agent_end, public terminal agent_end, and the session's asynchronous waitForIdle() drain. OMP 18.3.1 does not emit pi's agent_before_settle or agent_settled. Regressions exercise repeated busy consumption/ack followed by fresh idle mail, unread busy mail deferred through cleanup and fresh admission, native automatic continuations, approval resolution during work, Stop during cleanup, replacement, and admission crossing a busy transition. The shared suite retains rejected-send, reentrant acceptance, duplicate-loading and owning-root cases. These are adapter checks, not evidence of an authenticated model turn.

For real native proof, use an isolated coordinated TUI run with an authorized peer and the exact CLI/server/extension versions under investigation. Do not edit a live production hook, credentials, trust state or disable settings to obtain evidence. Record the managed launch arguments separately from manual file inspection; see activation.

1. Finish or block outstanding agent todos and confirm no native retry, approval prompt or pending input remains. End the normal model turn without a terminal worker report, active inbox --wait, polling loop or scheduled prompt. Observe at least two 30-second receiver waits with no new model turn.
2. Send one uniquely identified peer message using aether-internal send --to <receiver-run> --body <nonce> --idempotency-key <key>. Record the durable message ID, fresh admitted helper response, native API acceptance, model turn start, exact inbox body, and explicit aether-internal inbox --ack <ack_token>. After settlement, two more quiet observer waits must produce no extra model turn.
3. During ordinary foreground work, send another message. Let the agent read and acknowledge it before its final response. Repeat this sequence; each fresh idle snapshot must find no remaining mail and schedule no follow-up. In a separate round, leave mail unread through completion and require one deferred native wake after successful cleanup and fresh admission.
4. Test frozen batches separately: read a batch without acknowledging it, send another message, and read again. The old batch must repeat. Acknowledge its token only after handling it; the next batch must expose the new message. This is expected inbox behavior, not evidence of a missed native wake.
5. Exercise approval waits, Stop, protection/takeover and release, replacement, duplicate manual/managed loading, rejected native input and process exit. No busy or stale root may receive an automatic turn. Stop requires accepted human input to resume; releasing server control alone does not undo it.

Capture session/root identity, generation, message IDs, acknowledgement, helper admission, native turn boundaries and visible output. A final-looking assistant message, hidden inbox pointer, status reporter event, or empty inbox alone cannot identify the initiator. OMP's own todo/retry continuation may encounter legitimate integrator swarm-refresh context; that context does not start the turn. Name missing instrumentation and unexercised cases.

In the actual dashboard, open an owned swarm worker's terminal without touching control: it must remain a read-only mirror and request no write lease. Explicit Take control must acquire through its acknowledged control response; Release (Release control on the swarm page) must return to viewing, including after navigation. Check ordinary and integrator owner defaults, phone mirrors, foreign/protected runs and stale generations separately. The authoritative control/hold contract is in Run control, not duplicated by a test-only permission model.

The edge suite

internal/edge/edgetest runs edge remote access end to end in one process, under plain make test: a real edge (sign-in service and relay, serving its sign-in origin on localhost and its relay origin on 127.0.0.1) against a fake GitHub, real servers (sshd over a real store, with the edge agent) under each access policy, the real client dialer and the local gateway, and a browser played by an HTTP client with a cookie jar for the edge's own pages. A proxy in front of the edge records the control channels and, with the edge's own signing key, plays a compromised edge: it forges, replays, alters, drops and injects control messages and grants, answers the data socket of an open it forged, and routes a client's connection to another server. Tests read each server's store to check what an attack changed. edge.md describes the edge. It needs no Docker and no network:

go test -race ./internal/edge/edgetest/

Beside the suite, each package checks its own rules:

The integration suite's TestIntegrationUpgradeFromMainWithoutTheEdge starts this build on a database at main's schema version and a configuration without edge keys: any outbound HTTP request fails it, and its SSH key and tailnet members sign in as before.

Every request reaches the edge from 127.0.0.1, so each test moves its edge's clock forward to refill the per-address rate limits. Tests run in parallel, each with its own edge; the client calls that read the config directory from AETHER_CONFIG_DIR take turns.

Behind nginx

scripts/edge-nginx-test.sh, part of make test-scripts, runs TestBehindNginx (build tag nginx) when nginx is on PATH and skips otherwise. It renders packaging/nginx/aether-edge.conf.example for the host names localhost and 127.0.0.1 on a free port with a self-signed certificate, checks it with nginx -t, and runs a real nginx in front of the edge wired as aether-edge serve --proxy-listen wires it. A real server enrolls through nginx; a client signs in, claims, runs server.info, holds the relayed connection silent for 75 seconds (past nginx's default 60-second read timeout) and runs it again; and requests carrying a forged X-Forwarded-For are recorded under the address nginx saw. It takes about 80 seconds.

sh scripts/edge-nginx-test.sh

The edge image

scripts/edge-image-smoke.sh runs a built images/edge/Dockerfile image with fake GitHub credentials behind curl as its proxy, on a port published on 127.0.0.1. It checks that the image runs as a non-root numeric uid, has no shell and declares its HEALTHCHECK; that a start without configuration fails naming --signin-origin; that a GitHub-only configuration passes aether-edge healthcheck and exits 0 on SIGTERM within 30 seconds without logging the secret; and, through /v1/edge, that a replacement container on the same volume presents the same edge key fingerprint and one on a new volume a different one. It needs a container runtime, so it is not part of make test-scripts; the second argument names the runtime, docker by default:

make edge-image
sh scripts/edge-image-smoke.sh aether/edge:test

With podman, build with podman build --format docker -f images/edge/Dockerfile .: the default OCI format drops HEALTHCHECK, and the smoke test fails on it.

TestServeAsAContainerRunsIt in cmd/aether-edge covers the same start, healthcheck, stop and key checks on the binary without a runtime, with a read-only working directory, HOME and TMPDIR.

scripts/ci-classify-edge.sh decides from a pull request's changed paths whether the image can change. Its first case pattern lists every repository package aether-edge imports. scripts/ci-classify-edge-test.sh, part of make test-scripts, runs go list -deps ./cmd/aether-edge for linux/amd64 and linux/arm64 and fails when a package it imports is classified false. When the edge starts importing a new package, the test prints the line to add:

ci-classify-edge-test: aether-edge imports internal/<package> on linux/amd64, which ci-classify-edge.sh classifies false; add internal/<package>/* to its first case pattern

Add that pattern, then run sh scripts/ci-classify-edge-test.sh again.

The golang base image in images/edge/Dockerfile sets GOTOOLCHAIN=local, so the image builds with that image's Go whatever go.mod's toolchain line says. scripts/edge-go-version-test.sh, part of make test-scripts, fails when the two differ and names both versions:

edge-go-version-test: images/edge/Dockerfile builds with golang:1.26.8, but go.mod pins go1.26.9; change the golang tag and its @sha256 digest in images/edge/Dockerfile to 1.26.9 (or go.mod's toolchain line to go1.26.8)

The dashboard end-to-end suite

web/e2e/ drives the dashboard the way a person does: a Chromium browser on the static Next export the CLI embeds, talking to a real aether gui gateway, which proxies every call over a real SSH connection to a real aether-server. Both local and server-hosted paths dispatch through the shared internal/webgate; the server gateway's HTTPS/WhoIs boundary is covered by the integration row below and by the real-phone check. Playwright is the runner, pinned to an exact version in web/package.json.

Two projects share that one Chromium install. chromium uses the desktop descriptor and skips every *.mobile.spec.ts; mobile uses a Pixel-class descriptor - isMobile and hasTouch, so pointer: coarse matches and tap() sends real touch events - and runs those files alone. No spec runs under both. Run one with bunx playwright test --project=mobile from web/. A real phone reaches the same dashboard through the server gateway's URL. That path carries no browser token: Tailscale WhoIs identifies the source address on every request.

(cd web && bunx playwright install chromium)   # once, from the repo root
make test-e2e

make test-e2e builds the static dashboard export and both binaries first. The CLI serves the SPA out of its own embedded web/dist, so a stale binary would test a stale dashboard. The dashboard itself has no production Next server.

Administrator GitHub journeys

web/e2e/onboarding-github.spec.ts runs the real built dashboard and local SSH gateway against a test-only server process, real SQLite, Docker Environment exec, native signing setup, mirror storage and explicit revision adoption. It requires the repository's standard image, not the BusyBox image used by most dashboard scenarios: native Git, SSH signing, flock and timeout must actually work. Docker or a missing image is a failure, never a silent skip. From the repository root, with Go, Bun, Git, Docker and the project's build prerequisites installed:

docker info
docker build -f images/standard/Dockerfile -t aether-standard:ci .
(cd web && bun install --frozen-lockfile && bunx playwright install chromium)
make test-e2e E2E_ARGS='--project=chromium onboarding-github.spec.ts'

AETHER_E2E_STANDARD_IMAGE can name another locally available build of that image. make test-e2e rebuilds the embedded dashboard and gateway binaries. The harness additionally compiles the test-only server automatically using go test -c -tags=integration -o <scratch>/github-provider-server.test ./internal/server and runs it with -test.run=^TestGitHubBrowserProviderHarness$ -test.timeout=0. web/e2e/harness/server.ts supplies isolated AETHER_E2E_GITHUB_ROOT, AETHER_E2E_GITHUB_ADDR and AETHER_E2E_GITHUB_IMAGE; production binaries do not read those provider-fixture variables.

The external provider is substituted: a member-home gh fixture implements device approval/denial, Playwright intercepts the GitHub device page, the test-only server supplies GitHub HTTP metadata, and its fetch boundary uses real local Git repositories instead of GitHub HTTPS. No real GitHub secret is required, and no live GitHub approval, GitHub availability or production token-authenticated HTTPS transport is tested by this fixture. This is not an API-mocked dashboard: browser control requests still cross the gateway and assembled server. Separate live acceptance is needed for the provider boundary.

The unauthenticated gh auth status --json fixture returns exit status 0 with {"hosts":{}}, matching native gh's JSON-status behavior when no login exists. This lets the browser begin device authorization; a nonzero status remains a CLI failure, not evidence of a logged-out account.

The provider harness passes the built aether-server through its test-only AETHER_E2E_GITHUB_SERVER setting into Config.ServerBinary. The staged aether-internal helper must use the production dev-exec dispatcher, not the Go test executable running the provider harness.

Native pipe regressions exercise the real supervisor and Docker-multiplexed attachment: the device code and URL must arrive while the provider is still waiting for approval, with attachment stdin open. Approval and cancellation then produce their respective exit states:

go test ./internal/devexec ./internal/runtime \
  -run '^Test(SupervisorPipePublishesDeviceCodeBeforeApproval|ExecAttachmentPublishesDeviceCodeBeforeEOF)$'

Scenarios cover private read-only selection, inline acceptance pinned to both the configuration generation and the reviewed SHA, persisted connection reuse on the admin Agent step and in Settings, pagination to a second repository whose default branch is trunk, cancellation and denied authorization recovery, failed-fetch retry on the same workspace, and absent nonadmin controls plus forged-RPC denial. Successful checkpoints attach github-onboarding-device-code, github-onboarding-review-revision, github-onboarding-ready, github-settings-additional-repository-ready, github-cancellation-and-fetch-recovered and github-collaborator-settings PNG screenshots to the Playwright results/report. These are verification commands and expected coverage, not a claim that Docker or CI has passed.

Reviewed-revision regression coverage uses real Git: import candidate A, advance upstream to B and refresh without changing configuration generation, then submit adoption through the API with A as expected_commit. The stale request must fail before mutation, leaving accepted/base unchanged (or unset for an initial import) and B available as the candidate. Explicitly reviewing and submitting B must succeed. The UI retains the existing workspace on conflict and requires a fresh review rather than silently accepting B or repeating import. Generation alone is not a code-acceptance precondition.

The retained collaborator/native-terminal component coverage is in web/src/routes/onboarding/github-connect.test.tsx: typed login, missing/old/ broken gh remedies, probe retry, terminal incarnation changes, verbatim errors, account/signing fingerprint, CLI fallback and Back navigation. Its administrator case separately checks persisted connection reuse without opening a terminal. TestIntegrationGitHubConnect in internal/server/github_integration_test.go retains the native connection, signed commit and push path with a gh provider fixture; do not confuse that legitimate member setup with the new admin picker.

The browser suite owns behavior that jsdom cannot observe: actual hit testing, computed layout, responsive overflow, painted focus outlines and event ordering across document listeners. Component tests remain responsible for rendered roles, labels, state transitions, navigation and real gateway error text; a CSS class or source-pattern assertion is not a substitute for either layer.

The aether fixture (web/e2e/fixtures.ts) builds one stack per test and tears it down with everything it created:

• One aether-server child process on its own loopback SSH port, with a temporary data directory, AETHER_FAKE_AGENT="sh /workspace/agent.sh" in its environment and --standard-image busybox:1.36 - the tag internal/runtime's integration tests already pin, so a run of either suite warms the other's pull. Nothing is seeded into the store: the first identity to authenticate becomes the admin, which is what the wizard's Connect step does.
• One aether gui per member, each with its own HOME and AETHER_CONFIG_DIR, so the SSH key the Connect step generates, the known_hosts entry it writes, the saved link config and the member's persistent agent/configuration home all belong to that member and never touch the developer's own. PATH and SHELL are fixed too, because env.agents reports what is installed on this machine and that answer has to be the same on a laptop and on a runner.
• Real git repositories on disk, seeded with the agent.sh the fake harness runs.

Each gateway's git runs under an explicit GIT_SSH_COMMAND: OpenSSH resolves ~ from the password database rather than from HOME, so without it git would look for the member's key and known_hosts in the real user's home and fail host key verification.

The server under test is the shipped binary, so its containers carry only the production aether.managed label - there is no test label to sweep on. Teardown asks the server for its members and runs, then removes those containers by name, along with the aether/member-<member-id> images an environment save commits. A failed test keeps its scratch directory and attaches the server's output to the report.

Command palette performance

From web/, build the ordinary static export with bun run build. Use bun run build --profile for a separate React production-profiling export; do not substitute a development-server measurement for either.

Compare the same browser, viewport, pointer mode and datasets on both revisions: 50 and 500 runs, with short tasks and varied natural-prose tasks around 1,600 characters. Keep complete task bodies in the fixtures: the palette scores only each run's label, branch, agent (run.harness), workspace name and ID with cmdk's default scorer, and long tasks must not slow a keystroke past 16 ms. Record synthetic fixtures separately from live workspace data, and keep fixtures and raw traces outside the source tree.

Measure the first opening separately from at least 20 warm reopens. Use trusted keyboard and search-button input, repeated queries, backspacing and clearing. Report scorer time separately from input-to-results and opening latency, plus result counts, long tasks, requests and focus/selection behavior. An event-to-requestAnimationFrame-plus-timer measurement is a paint-opportunity proxy, not compositor latency. Compare ordinary and profiling builds separately.

Check full-text membership and ranking against cmdk's scorer, live run updates, offscreen keyboard selection, modal focus and Escape restoration. Filtering must not issue a search RPC. Score-disabled or hidden-Board ablations can locate a bottleneck; they do not prove the shipped behavior or its speed.

Scenarios

This inventory describes authored scenarios and their report attachments, not evidence that they have been executed or passed on a particular checkout.

account-sharing, activity, agent-outcome, board-card, candidate-delivery, development-browser/browser.spec.ts, development-terminal/shared-terminal.spec.ts, keyboard-focus, onboarding-agents, onboarding-first-run's launch scenario, remote-development-git/github.spec.ts, run-attach-retry, run-deep-link, run-evidence.spec.ts, run-files, run-provisioning, run-room.spec.ts, run-session, run-switch, templates, terminal-geometry, terminal-images, terminal-streaming and terminal-tools need a reachable Docker daemon and skip without one. onboarding-github, mission-candidate-review and remote-development-git/native.spec.ts also need Docker but fail without it instead of skipping. That skip is specific to the dashboard suite: make test-integration requires its real Docker setup and fails when Docker is unavailable. The rest need only git, except window-sizing, which needs neither: it starts a gateway of its own rather than taking the aether fixture, because the CLI half of update.check is answered on the member's own machine and no server is involved. The terminal image component tests separately pin File type/size validation, safe insertion without submission, native image-paste registration cleanup, and stale callback rejection after a terminal target remounts. The clipboard unit tests pin native Ctrl+Shift+V when the async clipboard API is denied.

The focused regressions own the editor and configuration edges without duplicating the browser smoke path:

• internal/memberhome/config_test.go covers member isolation, omitted-root reads, revision conflicts, import exclusions and unsafe-file preflight.
• internal/sshd/config_test.go covers authenticated own-member config RPC authorization and lifecycle behavior.
• web/src/routes/onboarding/agent-step.test.tsx covers the Agent step's setup, Run and Add agent paths and the prompt behind its Agent config files disclosure. web/src/routes/agents/agents.test.tsx covers Copy prompt writing exactly the prompt, a refused clipboard selecting it, Open remote files, and the section hidden without config.roots and config.import.
• web/src/components/agents/agent-setup.test.tsx covers the Standard and Enhanced comparison, agent.install with and without the adapter, the install failure, and the login wording.
• web/src/store/files.test.ts covers drafts surviving live-run cache invalidation and newer typing surviving an in-flight save.
• web/src/routes/files/home.test.tsx covers the Home tree: each kind of row's menu, also by right click, a symlink that does not open, Refresh, a credential file's refusal in the editor, naming in place with client and server refusals, F2 rename with the stem selected and the tab following, the delete dialog and its failure, uploads one at a time with one Replace question and the failure list, a dropped folder kept whole, the folder a drag rests on unfolding, a drop outside the tree refused, and a dragged row asking before it moves.
• web/src/routes/files/run-files.test.tsx covers a run's Files view (its tree, opening at a marked line, read-only, the reread that keeps the editor and the tree, the narrow layouts) and path links in the Session view: nothing asked until a path is pointed at or tapped, the two listings that answer, a path that is no file, a Markdown link to a repository path and a run whose checkout is gone. paths.test.ts covers what reads as a path, and web/src/routes/run/terminal-links.test.tsx a path link in a terminal, across a wrapped line.
• web/e2e/onboarding-configuration.spec.ts covers the prompt, its copy to the real clipboard and the way to Files; web/e2e/files-browser.mobile.spec.ts covers the narrow viewport tree/viewer round trip.

The phone project

web/playwright.config.ts defines two projects over the one Chromium install CI has. chromium runs every spec except *.mobile.spec.ts, and mobile runs only those, under Playwright's Pixel 7 descriptor: isMobile, hasTouch, a 412x839 viewport, a 2.625 device scale and a mobile user agent. Nothing runs twice, and no second browser engine is needed. iOS Safari is not covered - WebKit is not installed.

The installed window itself is not in the suite. Chromium exposes no display-mode override - not through emulateMedia, not through CDP's Emulation.setEmulatedMedia features, and not through the PWA domain, which headless does not carry - so a standalone window stays a manual check on a phone (docs/dashboard-frontend.md has that path). What the suite can prove is that Chrome is served a manifest it will accept, and that the shell needs nothing the browser's chrome was providing.

Mobile specs tap rather than click. locator.tap() dispatches touch events, and a control that answers only a mouse would still pass a click-driven test. They import test from web/e2e/mobile.ts, which attaches a full-page screenshot to every mobile test, passing or failing: a phone layout can be wrong while every DOM assertion holds, and a green run otherwise leaves nothing to look at.

shrinkToKeyboardHeight(page) in the same file takes the layout viewport down by 320px, what a keyboard leaves of a portrait phone, and returns the call that restores it. The shell asks for interactive-widget=resizes-content, so on a browser that honours it - Chrome and the Android WebView - that is what a real keyboard does to the layout viewport, and the helper reproduces the shape that ships rather than standing in for it. iOS Safari ignores the setting and shrinks only the visual viewport, so for that browser the helper proves the narrower claim: the shell survives a short screen. Playwright cannot raise a platform keyboard either way, so content stranded behind a real iOS keyboard stays a manual check on a phone (docs/dashboard-frontend.md has that path).

The phone specs need git; shell-drawer.mobile.spec.ts, run-views.mobile.spec.ts, run-room.mobile.spec.ts, run-evidence.mobile.spec.ts, terminal-phone.mobile.spec.ts and development-browser/browser.mobile.spec.ts also need Docker, because they open a real run, and skip without it. swarm.mobile.spec.ts drives its integrator's CLI through docker exec and fails without Docker. Run them alone against the binaries make build produced:

cd web && bunx playwright test --project=mobile

These specs run in make test-e2e and the dashboard-e2e job. The job uploads its playwright-report artifact on a pass as well as a failure, so the phone screenshots are on every run.

Reference screenshots from a Chromium touch audit use synthetic API data and a 132x43 terminal: terminal before/after and keyboard-height input, and short-screen page and dialog layouts. The audit exercised 390x844, 360x740 and 390x524 viewports, including the bottom actions of long pages and dialogs. These images show browser layout, not a physical keyboard or a live vendor session.

Adding a step to the wizard

web/e2e/pages/wizard.ts is the page-object layer, and a new wizard step is one class and one field. Give the class the aria-label of the step's <section> and the actions that step offers, add its label to stepNames in the order the header lists it, and hang it off OnboardingWizard. Every locator a step builds is scoped to its own section, so nothing else in the suite changes.

Availability regression commands

These focused commands map the retained regressions in the availability paths. They are useful before running the full gates; the package tests still belong in the normal CI jobs.

# Scheduler wait errors, cancellation, and durable terminal cleanup.
go test -race ./internal/scheduler -run \
  'TestSuperviseWaitRetriesTransportErrorUntilExit|TestSuperviseWaitCancellationDuringRetryLeavesRunLive|TestSuperviseTerminalRetriesTransportErrorUntilExit|TestExitedTerminalCleanupRetainsStateForRetry'

# PTY session isolation, cancellation, and stopped-session replay.
go test -race ./internal/ptyhost -run \
  'TestActiveSessionsDoesNotBlockUnrelatedAttach|TestReserveDoesNotBlockUnrelatedStart|TestAttachContextCancel|TestAttachDrainHonorsContext|TestStoppedSessionClosesAttachmentAndPreservesReplay'

# Direct forwarding: half-close drains; full disconnect cancels.
go test -race ./internal/sshd -run \
  'TestDirectTCPIPOwnerEchoAndHalfClose|TestDirectTCPIPFullDisconnectReleasesBackend|TestDirectTCPIPDisconnectCancelsAddressResolution'

# Scalar cost summaries preserve metered/unmetered budget semantics.
go test -race ./internal/store ./internal/cost -run \
  'TestSummarizeRunCostsMatchesRollupSemantics|TestBudgetReflectsCostHistoryAcrossUpdatesAndWorkspaces|TestUnmeteredSpendNeverCountsTowardTheCap'

# Real git: ignored-tree pruning, live ignore changes, and pack cancellation.
go test -race -tags integration ./internal/gitengine -run \
  'TestDiffWatch|TestUploadPackReturnsOnCtxCancel'

# Retained transcript expiry, restart reuse, and submission integrity.
go test -race ./internal/evidence -run \
  'TestExpiredPrefixCaptureSurvivesRestartAndHistoryRemoval|TestTranscriptCaptureDistinguishesPrefixAndCopyCap|TestLegacyTranscriptMarkersMigrateWithoutRelaxingCap|TestPrefixCaptureReadFailureNeverPublishesPartialArtifact'

# Durable recording boundaries, legacy frontiers and explicit deletion.
go test -race ./internal/ptyhost ./internal/acphost ./internal/scheduler

Durable-history acceptance uses real cast segments, JSONL logs and Git sidecars. Cross the former size, count and age boundaries, restart, then read the earliest recording and recorded diff interval. Exercise checkout cleanup separately: it must remove the worktree without removing recordings or recorded trees. Explicit run deletion must still remove those histories. Archive old runs and swarms and verify they remain restorable without an automatic deletion deadline.

Already-trimmed recordings retain their original absolute positions and missing-prefix metadata. Check authenticated expired cursors, full-item lookups, live approvals and evidence provenance against those existing gaps. Never substitute empty output or restart sequence numbering.

The Linux replay regression lowers the descriptor budget only in an isolated helper process and streams 257 segments without discarding old ones. Check exact bytes, compact screen position and finite replay under concurrent append/rotation. Explicit deletion may interrupt an unopened sealed segment, but must not silently skip it. Archived SSH acknowledgments and download headers continue reporting pre-existing gaps without injecting marker bytes.

TestCheckpointLongLineageRemainsWritable keeps 40,000 real recording files whose old checkpoint metadata would exceed 8 MiB, then verifies compact checkpoint publication, rotation, restart, old cursors and continued output. History-discovery regressions cross the former directory-entry limit and advance through bounded pages of empty segments without a stuck cursor.

The Docker runtime regressions must run against a reachable, real Docker daemon; the in-process E2E runtime is not a substitute for these checks:

docker info
go test -race -tags integration ./internal/runtime -run \
  'TestDockerStartCancelDuringSetup|TestDockerStartKillsAfterLostStartReply|TestDockerStartTwiceSkipsSetup|TestDockerWaitBeforeStart|TestDockerInitReapsOrphanedDescendants'

make test-integration is the merge gate for the complete real-Docker, real-git integration suite. It must fail rather than silently pass when Docker is unavailable; run it in CI when the local host has no daemon.

Failure-table coverage

Every row of the design spec's failure table has at least one covering scenario; rows not exercised end to end are pinned by unit tests at the layer that owns them.

Rules

• Prefer a scenario on the real user path over a pile of internal tests; never restate a behaviour already proven at another layer.
• Bug fixes start with an E2E reproduction.
• Keep the suite fast enough to gate merges: agents are scripted and deterministic, containers are seconds-lived, and every test sweeps and checks for leaked containers via its aether.test label.
• Never let a test depend on real time passing. A cache or a deadline takes an injectable clock the test winds by hand; a sleep or a tiny TTL is a test that fails on whichever platform has the coarsest timer, and Windows' is coarse enough that two reads of the clock can return the same instant.
• Behaviour that differs by platform gets a test per platform, not one that skips. The client packages run on a Windows runner too (.github/workflows/ci.yml), so a test whose subject refuses on Windows - the self-update swap, say - goes in a //go:build !windows file with a _windows_test.go counterpart asserting the refusal. Assert the status before the body: an error envelope decodes into a result struct just as happily, and a test that reads only the body can pass on the platform that refused.