Testing and the E2E scenario suite

Layers, per the design spec's testing strategy:

• Unit tests live beside their packages and run with make test (race detector on). Permission matrices, budget math, configuration import and file revision rules, tailnet auth edge cases, scheduler transitions, and the local gateway's own behaviors are proven there, once, and the E2E suite does not restate them. Role changes belong to the same layer: internal/sshd/role_test.go and internal/sshd/permissions_test.go own promotion, demotion, the last-admin guard and what each role may do, and the SPA's half of it is in its rendered route tests. The multi-member E2E row below joins members and administers them; it does not re-prove the matrix.
• Integration/E2E tests are behind the integration build tag and run with make test-integration (real Docker, real git), which covers only the packages carrying integration-tagged tests. INTEGRATION_PKGS narrows that to one package and INTEGRATION_SKIP leaves some out, as in make test-integration INTEGRATION_PKGS=./internal/server. CI runs them on every PR from .github/workflows/ci.yml: the integration matrix shards them by package, one job each for internal/server and internal/scheduler and one for the rest, and the smoke job runs internal/harness on the images it builds. Those jobs are the merge gate the E2E suite owns.
• Dashboard component tests live beside their components in web/src/ and run with bun run test from web/ (vitest in jsdom). CI runs them in the dashboard job. jsdom has no layout, so web/src/test/setup.ts answers every media query with false and a component renders its widest branch; atViewport (web/src/test/viewport.ts) puts one test on one screen instead - width and pointer queries answer for it, window.innerWidth/innerHeight report it, and the returned resize fires change where an answer moved and resize on the window. It decides which branch renders and nothing more: real layout belongs to the browser suite below. setup.ts also empties the rules of every stylesheet whose <style> element has been detached. jsdom unregisters a sheet only while that element is still connected, and xterm discards its three style elements with the terminal around them, so each terminal a file mounts strands three sheets of roughly 800 rules. The entries themselves stay in document.styleSheets and their count still grows; emptying their rules is what keeps the cost flat, because getComputedStyle - which every testing-library visibility query runs - re-cascades only the rules that are still there.
• Dashboard end-to-end tests live in web/e2e/ and run with make test-e2e: a real browser driving the static Next export embedded by the shipped binary, through a real aether gui gateway and a real aether-server. They own the paths a person walks in the dashboard, which no Go test and no jsdom test reaches. CI runs them in the dashboard-e2e job.

Windows CI runs TestInstallDesktopWindowsPreservesCLI in internal/localops: install and reinstall must preserve the CLI and unrelated files in the documented CLI directory and detect only the desktop directory as an installed app. TestShellLinkLaunchesNativeConsumer exercises Windows' real shortcut launcher with a target path containing spaces, an ampersand, and non-ASCII characters, and checks the launched process's working directory. These checks run before desktop packaging.

The Windows install workflow runs scripts/install-test.ps1 under Windows PowerShell 5.1 and PowerShell 7. Those scenarios cover checksum rejection before replacement, upgrade and locked-file boundaries, PATH preservation, CLI-only installation, unsupported releases, and desktop-build failures. They use temporary files and restore the user's environment after running.

The same workflow builds the real CLI with the release's Windows metadata and embedded dashboard, removes the hosted runner's inherited exclusions, and enables Defender realtime, script, archive, and first-seen cloud scanning before running scripts/install-smoke.ps1. A local release mirror serves those exact bytes and their checksum so the scenario tests the checkout, not the latest published release. It installs and rebuilds the desktop, verifies the unchanged CLI, and launches the installed Start Menu shortcut with the pre-install PATH. Playwright checks that window's onboarding screen and saves a screenshot. Windows PowerShell uses system Node; PowerShell 7 hides system Node to exercise the verified private download. Defender scans the download and install tree without exclusions or disabled remediation. The gate rejects new detections even when Defender has already remediated them, and checks that installation changed neither protection settings nor exclusions. This is a detection gate, not a guarantee that an unsigned release will never receive a false positive on another machine.

Local configuration in tests

Tests that read or write the linked-server config through cli.Load, cli.Save, or gateway handlers must call internal/testhome.Isolate(t). It sets AETHER_CONFIG_DIR, the platform home/config variables, and clears SSH_AUTH_SOCK. Setting only XDG_CONFIG_HOME and AppData leaves the real macOS config at ~/Library/Application Support/aether/config.json exposed. The gateway regression in internal/localgw/config_isolation_test.go runs config refresh and repository linking against a temporary user config and checks that its contents and modification time stay unchanged, including when the test process inherits AETHER_CONFIG_DIR.

The E2E scenario suite

internal/server's *_integration_test.go files are the owned end-to-end suite: every scenario drives the fully wired server (server.New) over real SSH, real git transport, and - when the daemon is reachable - real Docker containers. pickRuntime falls back to the in-process e2eRuntime (e2eruntime_test.go) on hosts without Docker; the two host-half coordination scenarios force it because their agents must reach container surfaces from the test process, and the container coordination scenario skips without a daemon rather than falling back. Scenarios:

The chaos scenarios

chaos_reboot_integration_test.go is the one place the suite runs aether-server as a child process. An in-process server cannot be SIGKILLed, and the whole point of that row is that nothing on the shutdown path runs: the next boot only sees what SQLite and git had already made durable. The child binary is built once per test binary, the store is seeded before startup, and it binds a reserved loopback port so a restart can claim the same address. It always builds its own Docker runtime, so these scenarios skip without a reachable daemon rather than falling back, and they address containers by the name the runtime derives from the run ID.

Disk pressure is driven through configuration, never by filling the host disk: --min-free-disk above what the machine has free makes the real statfs path refuse for real, and --checkout-ttl turned down makes the real GC sweep on the next boot. No injected filesystem, no fake statfs.

The SSH-drop boundary is fuzzed where it lives, in internal/ptyhost (drop_fuzz_test.go): the connection is cut at every byte offset and the agent's stdin must always be an exact prefix of what the transport delivered, which catches a byte from past the cut as well as any reorder or duplication. Cutting the stream cannot reach the other half of that row - the read loop stops at the first error and never asks a dead connection for more - so the case where the attach unwinds first and the socket coughs up a straggler afterwards has its own scenario in the same file.

The deterministic fake agent is the scheduler's fake harness: its argv comes from AETHER_FAKE_AGENT at launch (typically sh /workspace/agent.sh {task}, with the script committed to the seed repo and dispatching on the task). On the fallback runtime the same behaviours are registered per task key via e2eRuntime.script.

Three server.Config fields exist for this suite: WhoIs overrides tailnet identity resolution so join and fallback scenarios need no real tailnet, Harnesses overrides registry argv templates so a registered harness (with its real profile root and credential mounts) can run a scripted agent - the first two double as deployment wiring - and ServerBinary names the executable staged for the CLI and optional MCP bridge.

The container coordination scenario

coordination_container_integration_test.go proves the half of docs/mcp-bridge.md that an assertion on a container spec cannot: that the mounts a run is given are real, and that the agent holding them can use them. Two things make it possible.

The staged bridge has to be a binary that has the mcp subcommand, which under go test /proc/self/exe is not. So the scenario points ServerBinary at an aether-server it builds - the same one the chaos scenarios run as a child process.

The scenario uses the shipped claude profile with a non-root fixture agent from internal/server/testdata/coordagent. It explicitly invokes the optional bridge from the canonical executable mount; no automatic MCP registration is required. The fixture reports directory modes, read-only mounts from the kernel's mount table, EROFS on attempted writes, and tool results over a real attach. The daemon's mount view is checked alongside those observations.

TestIntegrationMissionCompositionInDocker composes mission dispatch, proactive coordination, accepted retained submissions, combined verification, human approval, and exact delivery. It also checks failed verification, stale-target rejection, and integrator replacement. Its claude, pi, and omp executables are scripted fixtures, not genuine vendor-agent runs. web/e2e/mission-candidate-review.spec.ts drives launch, progress, worker control, and mission candidate preparation in a real browser and attaches a successful screenshot for visual inspection.

The container user is the test process's own uid:gid unless that is root: the scheduler chowns the run checkout and the member home to the container user before creating the container, and an unprivileged test process can only chown to itself.

The real-harness smoke tests

internal/harness/smoke_integration_test.go launches the vendors' actual CLIs and checks that the argv Aether ships is still the argv they accept. Nothing else catches a vendor renaming a flag or refusing a combination it used to allow: a change like that breaks every run of that harness and no amount of internal testing sees it coming.

TestSmokeHeadlessNoLogin is the one that runs in CI. It launches each headless template with no credentials at all and requires the run to fail for want of a login, never for want of a parseable command line - reaching the provider is proof the CLI accepted the flags. The smoke job builds images/standard/Dockerfile, then images/smoke/Dockerfile on top of it to add the agent CLIs at whatever version their vendors ship that day, and points the gate variables at the result.

To run it locally, build the same image and name it:

docker build -f images/standard/Dockerfile -t aether-standard:local .
docker build -f images/smoke/Dockerfile --build-arg BASE=aether-standard:local \
  -t aether-smoke:local .
AETHER_SMOKE_IMAGE_CLAUDE_NOLOGIN=aether-smoke:local \
AETHER_SMOKE_IMAGE_CODEX_NOLOGIN=aether-smoke:local \
AETHER_SMOKE_IMAGE_OPENCODE_NOLOGIN=aether-smoke:local \
  go test -tags integration -run TestSmokeHeadlessNoLogin ./internal/harness/

The _NOLOGIN images must carry no credentials. The other smoke tests - TestSmokeClaude, TestSmokeOpencode, TestSmokeCodexFlags - drive the agent far enough to produce output, so they need an image that does carry that harness's login state, named by AETHER_SMOKE_IMAGE_<NAME>. CI has no such credentials, so those stay a manual check. Every one of them skips when its variable is unset, and the no-login test skips as a whole rather than reporting a pass with nothing run.

pi and omp have no smoke test: neither is in the smoke image, so nothing pins their templates. Adding one is an install line and a map entry each.

The status extension those two load, internal/agentstatus/status.ts, is the one shipped file the Go build never executes. internal/agentstatus runs it under bun against a recording stand-in for the server binary, so a turn's reports are proven to come out in order and to end exactly once. Those tests skip where bun is not installed.

The dashboard end-to-end suite

web/e2e/ drives the dashboard the way a person does: a Chromium browser on the static Next export the CLI embeds, talking to a real aether gui gateway, which proxies every call over a real SSH connection to a real aether-server. Both local and server-hosted paths dispatch through the shared internal/webgate; the server gateway's HTTPS/WhoIs boundary is covered by the integration row below and by the real-phone check. Playwright is the runner, pinned to an exact version in web/package.json.

Two projects share that one Chromium install. chromium uses the desktop descriptor and skips every *.mobile.spec.ts; mobile uses a Pixel-class descriptor - isMobile and hasTouch, so pointer: coarse matches and tap() sends real touch events - and runs those files alone. No spec runs under both. Run one with bunx playwright test --project=mobile from web/. A real phone reaches the same dashboard through the server gateway's URL. That path carries no browser token: Tailscale WhoIs identifies the source address on every request.

(cd web && bunx playwright install chromium)   # once, from the repo root
make test-e2e

make test-e2e builds the static dashboard export and both binaries first. The CLI serves the SPA out of its own embedded web/dist, so a stale binary would test a stale dashboard. The dashboard itself has no production Next server.

The browser suite owns behavior that jsdom cannot observe: actual hit testing, computed layout, responsive overflow, painted focus outlines and event ordering across document listeners. Component tests remain responsible for rendered roles, labels, state transitions, navigation and real gateway error text; a CSS class or source-pattern assertion is not a substitute for either layer.

The aether fixture (web/e2e/fixtures.ts) builds one stack per test and tears it down with everything it created:

• One aether-server child process on its own loopback SSH port, with a temporary data directory, AETHER_FAKE_AGENT="sh /workspace/agent.sh" in its environment and --standard-image busybox:1.36 - the tag internal/runtime's integration tests already pin, so a run of either suite warms the other's pull. Nothing is seeded into the store: the first identity to authenticate becomes the admin, which is what the wizard's Link step does.
• One aether gui per member, each with its own HOME and AETHER_CONFIG_DIR, so the SSH key the Link step generates, the known_hosts entry it writes, the saved link config and the member's persistent agent/configuration home all belong to that member and never touch the developer's own. PATH and SHELL are fixed too, because env.harnesses reports what is installed on this machine and that answer has to be the same on a laptop and on a runner.
• Real git repositories on disk, seeded with the agent.sh the fake harness runs.

Each gateway's git runs under an explicit GIT_SSH_COMMAND: OpenSSH resolves ~ from the password database rather than from HOME, so without it git would look for the member's key and known_hosts in the real user's home and fail host key verification.

The server under test is the shipped binary, so its containers carry only the production aether.managed label - there is no test label to sweep on. Teardown asks the server for its members and runs, then removes those containers by name, along with the aether/member-<member-id> images an environment save commits. A failed test keeps its scratch directory and attaches the server's output to the report.

Scenarios

board-card, keyboard-focus, onboarding-agents, onboarding-github, onboarding-first-run's launch scenario, run-attach-retry, run-deep-link, run-provisioning, run-switch, run-views.mobile.spec.ts, shell-drawer.mobile.spec.ts, terminal-geometry, terminal-images and terminal-tools need a reachable Docker daemon and skip without one. That skip is specific to the dashboard suite: make test-integration requires its real Docker setup and fails when Docker is unavailable. The rest need only git, except window-sizing, which needs neither: it starts a gateway of its own rather than taking the aether fixture, because the CLI half of update.check is answered on the member's own machine and no server is involved. The terminal image component tests separately pin File type/size validation, safe insertion without submission, native image-paste registration cleanup, and stale callback rejection after a terminal target remounts. The clipboard unit tests pin native Ctrl+Shift+V when the async clipboard API is denied.

The focused regressions own the editor and configuration edges without duplicating the browser smoke path:

• internal/memberhome/config_test.go covers member isolation, omitted-root reads, revision conflicts, import exclusions and unsafe-file preflight.
• internal/sshd/config_test.go covers authenticated own-member config RPC authorization and lifecycle behavior.
• web/src/routes/onboarding/agents-step.test.tsx covers the import UI's explicit action, destination selection and excluded-file reporting.
• web/src/store/files.test.ts covers drafts surviving live-run cache invalidation and newer typing surviving an in-flight save.
• web/e2e/onboarding-configuration.spec.ts covers the browser import and config read/write path; web/e2e/files-browser.mobile.spec.ts covers the narrow viewport tree/viewer round trip.

The phone project

web/playwright.config.ts defines two projects over the one Chromium install CI has. chromium runs every spec except *.mobile.spec.ts, and mobile runs only those, under Playwright's Pixel 7 descriptor: isMobile, hasTouch, a 412x839 viewport, a 2.625 device scale and a mobile user agent. Nothing runs twice, and no second browser engine is needed. iOS Safari is not covered - WebKit is not installed.

The installed window itself is not in the suite. Chromium exposes no display-mode override - not through emulateMedia, not through CDP's Emulation.setEmulatedMedia features, and not through the PWA domain, which headless does not carry - so a standalone window stays a manual check on a phone (docs/dashboard-frontend.md has that path). What the suite can prove is that Chrome is served a manifest it will accept, and that the shell needs nothing the browser's chrome was providing.

Mobile specs tap rather than click. locator.tap() dispatches touch events, and a control that answers only a mouse would still pass a click-driven test. They import test from web/e2e/mobile.ts, which attaches a full-page screenshot to every mobile test, passing or failing: a phone layout can be wrong while every DOM assertion holds, and a green run otherwise leaves nothing to look at.

shrinkToKeyboardHeight(page) in the same file takes the layout viewport down by 320px, what a keyboard leaves of a portrait phone, and returns the call that restores it. The shell asks for interactive-widget=resizes-content, so on a browser that honours it - Chrome and the Android WebView - that is what a real keyboard does to the layout viewport, and the helper reproduces the shape that ships rather than standing in for it. iOS Safari ignores the setting and shrinks only the visual viewport, so for that browser the helper proves the narrower claim: the shell survives a short screen. Playwright cannot raise a platform keyboard either way, so content stranded behind a real iOS keyboard stays a manual check on a phone (docs/dashboard-frontend.md has that path).

The phone specs need git; shell-drawer.mobile.spec.ts, run-views.mobile.spec.ts and terminal-phone.mobile.spec.ts also need Docker, because they open a real run, and skip without it. Run them alone against the binaries make build produced:

cd web && bunx playwright test --project=mobile

They add about 15 seconds to make test-e2e and to the dashboard-e2e job, which stays inside the suite's 30-minute globalTimeout unchanged. That job uploads its playwright-report artifact on a pass as well as a failure, so the phone screenshots are on every run.

Adding a step to the wizard

web/e2e/pages/wizard.ts is the page-object layer, and a new wizard step is one class and one field. Give the class the aria-label of the step's <section> and the actions that step offers, add its label to stepNames in the order the header lists it, and hang it off OnboardingWizard. Every locator a step builds is scoped to its own section, so nothing else in the suite changes.

Availability regression commands

These focused commands map the retained regressions in the availability paths. They are useful before running the full gates; the package tests still belong in the normal CI jobs.

# Scheduler wait errors, cancellation, and durable terminal cleanup.
go test -race ./internal/scheduler -run \
  'TestSuperviseWaitRetriesTransportErrorUntilExit|TestSuperviseWaitCancellationDuringRetryLeavesRunLive|TestSuperviseTerminalRetriesTransportErrorUntilExit|TestExitedTerminalCleanupRetainsStateForRetry'

# PTY session isolation, cancellation, and stopped-session replay.
go test -race ./internal/ptyhost -run \
  'TestActiveSessionsDoesNotBlockUnrelatedAttach|TestReserveDoesNotBlockUnrelatedStart|TestAttachContextCancel|TestAttachDrainHonorsContext|TestStoppedSessionClosesAttachmentAndPreservesReplay'

# Direct forwarding: half-close drains; full disconnect cancels.
go test -race ./internal/sshd -run \
  'TestDirectTCPIPOwnerEchoAndHalfClose|TestDirectTCPIPFullDisconnectReleasesBackend|TestDirectTCPIPDisconnectCancelsAddressResolution'

# Scalar cost summaries preserve metered/unmetered budget semantics.
go test -race ./internal/store ./internal/cost -run \
  'TestSummarizeRunCostsMatchesRollupSemantics|TestBudgetReflectsCostHistoryAcrossUpdatesAndWorkspaces|TestUnmeteredSpendNeverCountsTowardTheCap'

# Real git: ignored-tree pruning, live ignore changes, and pack cancellation.
go test -race -tags integration ./internal/gitengine -run \
  'TestDiffWatch|TestUploadPackReturnsOnCtxCancel'

The Docker runtime regressions must run against a reachable, real Docker daemon; the in-process E2E runtime is not a substitute for these checks:

docker info
go test -race -tags integration ./internal/runtime -run \
  'TestDockerStartCancelDuringSetup|TestDockerStartKillsAfterLostStartReply|TestDockerStartTwiceSkipsSetup|TestDockerWaitBeforeStart|TestDockerInitReapsOrphanedDescendants'

make test-integration is the merge gate for the complete real-Docker, real-git integration suite. It must fail rather than silently pass when Docker is unavailable; run it in CI when the local host has no daemon.

Failure-table coverage

Every row of the design spec's failure table has at least one covering scenario; rows not exercised end to end are pinned by unit tests at the layer that owns them.

Rules

• Prefer a scenario on the real user path over a pile of internal tests; never restate a behaviour already proven at another layer.
• Bug fixes start with an E2E reproduction.
• Keep the suite fast enough to gate merges: agents are scripted and deterministic, containers are seconds-lived, and every test sweeps and checks for leaked containers via its aether.test label.
• Never let a test depend on real time passing. A cache or a deadline takes an injectable clock the test winds by hand; a sleep or a tiny TTL is a test that fails on whichever platform has the coarsest timer, and Windows' is coarse enough that two reads of the clock can return the same instant.
• Behaviour that differs by platform gets a test per platform, not one that skips. The client packages run on a Windows runner too (.github/workflows/ci.yml), so a test whose subject refuses on Windows - the self-update swap, say - goes in a //go:build !windows file with a _windows_test.go counterpart asserting the refusal. Assert the status before the body: an error envelope decodes into a result struct just as happily, and a test that reads only the body can pass on the platform that refused.