Scout

Writing scenarios

The .scout.md format: one file per feature, plain-language flows, browser permissions, multi-tab follows, show/hide (opacity) checks, and console/network assertions — versioned and reviewed like the rest of your tests.

Scenarios live as markdown under .scout/specs/**/*.scout.md — one file per feature/component, versioned and reviewed like the rest of your tests. The markdown is the source of truth and a pure input: a run never writes back to it. Status and last-run derive from .scout/runs/ instead, so the spec diff only ever reflects an intent change, never run noise.

---
feature: Paywall          # optional; defaults to the filename
profile: anon             # default auth profile for scenarios below
tags: [monetization]      # optional
viewports: [mobile]       # optional; screen sizes each scenario runs in
---

## Free user hits paywall on ep 3
Open ep 3 of series X without login; paywall appears with a signup CTA.

## Subscriber bypasses paywall
profile: qa               # per-scenario override (also: notes, tags, viewports)
viewports: [mobile, desktop]

Logged-in subscriber opens ep 3; plays with no paywall.

You state what a user can do and what must be true — never how to click it. That is the whole point: the sentence is the intent you hold the product to, and the agent discovers the real elements at run time so the wording survives a redesign.

Rules

  • Each ## heading is one scenario. Its logical slug is <file>/<scenario> (e.g. paywall/free-user-hits-paywall-on-ep-3) and must be unique across the suite.
  • Frontmatter (YAML, optional): feature (defaults to the filename), profile (default auth profile for the file), tags, viewports.
  • Per-scenario overrides: profile:, notes:, tags:, viewports:, the permission keys (grantPermissions:, denyPermissions:, geolocation:), cookies:, storage:, and device: placed immediately under a heading (before the prose) override the file-level defaults.
  • Body = flow + expected behavior, in plain language. Describe what the user does and what must (or must not) be true. No CSS selectors, no Playwright code — the agent discovers the real elements at run time and records them.
  • A .scout.md whose every ## lives inside a fenced ``` block parses as zero scenarios (that’s how example.scout.md documents the format without polluting the suite).
  • Duplicate headings in a file, or a scenario with no body text, are hard errors.

Viewports (screen sizes)

Each scenario runs in one or more named viewports. List more than one and the scenario fans out into independent verification units — each viewport gets its own recorded script (<slug>@<viewport>.json), its own verdict, and its own demo video. That is how you cover responsive behavior: the mobile hamburger and the desktop nav are different flows, verified separately.

Three viewports are built in, usable with no config:

  • mobile — iPhone 13 emulation (touch + mobile UA), pinned to 390×844.
  • desktop — 1280×800, no touch.
  • tablet — iPad Mini (768×1024, touch).

A per-scenario viewports: list replaces the file-level one (it does not merge like tags) — each scenario explicitly chooses its sizes. Declare none anywhere and scout uses the config’s defaultViewport (mobile).

---
feature: Navigation
viewports: [mobile, desktop]   # file default: every scenario runs in both
---

## Primary nav reaches Pricing
Open the home page and reach the "Pricing" link from the main navigation.

## Hamburger opens the drawer
viewports: [mobile]            # only meaningful on small screens

Open the home page; tapping the menu button reveals the navigation drawer.

Names are limited to [a-z0-9-] (they become script-file tokens); a name not in the registry fails the run with a clear error. Override or add viewports in scout.config.json, and force one ad-hoc with scout go --viewport <name> (that run never persists a script). The demo video follows the viewport — a desktop scenario records a landscape clip.

Browser permissions

Flows that trigger a native permission prompt (geolocation, notifications, camera, microphone) can’t be driven by the agent — the prompt lives outside the page. Declare a policy and scout sets it at browser launch, for both the AI run and replay. Three states per permission: not declared → native behavior (untouched); grantPermissions → granted; denyPermissions → blocked, so no prompt appears.

Set it in the frontmatter (file default) and/or per scenario (overrides merge per-axis, and grant wins over deny):

---
feature: Store Locator
denyPermissions: [geolocation]      # file default: never prompt for location
---

## Search falls back to manual when location is blocked
Open the store locator, search for "Merate", confirm results appear.

## Nearby stores with a fixed location
grantPermissions: geolocation       # this scenario grants instead
geolocation: 45.69, 9.43            # required when geolocation is granted (lat, lng)

Open the store locator; the nearest store to the coordinates is shown.

Allowed: geolocation, notifications, camera, microphone, clipboard-read, clipboard-write, midi — an unknown name is a hard error. Granting geolocation requires coordinates (and coordinates imply granting it). denyPermissions matters mostly in headed runs (headless denies silently); grantPermissions and a granted geolocation change behavior in CI too.

Cookies

When a flow’s precondition is a cookie — forcing a server-side experiment variant, a feature flag, a consent state — declare it and scout seeds it into the browser before the first navigation, for both the AI run and replay. It’s a context-creation parameter (like storageState), never a recorded step, so replay stays deterministic and the agent never needs a “set cookie” tool.

Two forms. In the frontmatter (file default) or a profile (shared base in scout.config.json), cookies: is a list of objects — the place for attributes:

---
feature: Checkout
cookies:
  - name: hn_checkout_variant
    value: A                 # file default: variant A
  - name: consent
    value: "yes"
    httpOnly: true
    sameSite: Lax            # Strict | Lax | None
---

## Default variant
Open checkout; the file-level variant A is in effect.

## Force variant C
cookies: hn_checkout_variant=C   # per-scenario override: inline name=value

Open checkout; variant C is forced.
  • Merge by name: a profile’s cookies are the base; the file frontmatter and then the per-scenario override win, keyed by cookie name. The per-## override is the terse inline name=value[, n2=v2] form (no attributes — those belong in the frontmatter/profile).
  • name and value are required; domain, path, expires (unix seconds), httpOnly, secure, sameSite are optional. domain/path default to the host of baseUrl and /.
  • Secrets: a value may use the $ENV:VAR placeholder (e.g. cookies: session=$ENV:SESSION_TOKEN) — resolved at launch, so the secret never lands in the committed spec or the agent’s context. Only value is resolved.
  • Fail-fast: an unknown field, a bad sameSite, or a missing env var is a hard error — a silently skipped cookie precondition would produce a misleading verdict.

Local & session storage

Some features are gated not by a cookie or the server, but by the browser’s own web storage — an open-count threshold, a “you’ve seen this” flag, a dismissed prompt. The agent can’t set storage (it has no page.evaluate tool), so those scenarios used to stall at partial. Declare a storage: seed and scout applies it before the app loads, for both the AI run and replay. Like cookies/storageState, it’s a context-creation parameter, never a recorded step — replay re-resolves it fresh from the spec, so it stays deterministic and the agent never needs a “set storage” tool.

It seeds both localStorage and sessionStorage (the latter is something storageState can’t carry) via an init-script that runs before any page script, and it can remove keys to guarantee a clean precondition.

Two forms. In the frontmatter (file default) or a profile (shared base in scout.config.json), storage: is an object with local, session and/or remove:

---
feature: PWA install prompt
storage:
  local:
    hn_app_open_count: "2"      # file default: two prior opens
  remove:
    - hn_pwa_prompt_dismissed   # start from a non-dismissed state
---

## Below the threshold — no prompt yet
Open the app; the install prompt does not appear.

## At the threshold — prompt appears
storage: local.hn_app_open_count=3, remove=hn_other_flag

Open the app; the install prompt appears.
  • Inline override form: the per-## override is a single line of comma-separated tokens — local.<key>=<value>, session.<key>=<value>, or remove=<key>. The namespace is spelled inline because a heading override line can’t carry a nested YAML object; nested objects belong in the frontmatter/profile.
  • Merge: a profile’s storage is the base; the file frontmatter and then the per-scenario override win, per key per namespace. remove lists concatenate and dedupe across all levels.
  • remove clears both namespaces: a removed key is dropped from localStorage and sessionStorage, so a stale value never leaks into the run. Removals apply before the seed, so a declared value always wins over a removal of the same key.
  • Secrets: a value may use the $ENV:VAR placeholder — resolved at launch, so the secret never lands in the committed spec or the agent’s context.
  • Fail-fast: an unknown field (only local/session/remove are allowed), a non-string value, or a malformed inline token is a hard error — a silently skipped storage precondition would produce a misleading verdict.

Device / user-agent emulation

Some UI is gated on device or user-agent detection — an “Add to Home Screen” sheet that only renders under an iOS-Safari UA, a layout branch that keys off touch support. By default scout launches desktop Chromium with its own UA, so those flows can never pass. Declare a device: and scout emulates it at browser launch, for both the AI run and replay. Like cookies/storage/storageState, it’s a context-creation parameter, never a recorded step — replay re-reads the frontmatter, so it stays deterministic.

device names a Playwright device descriptor (e.g. iPhone 14, Pixel 7). Individual fields — userAgent, viewport ({ width, height }), deviceScaleFactor, isMobile, hasTouch — compose on top of (or without) the named device; an explicit field always wins over the device’s value.

Two forms. In the frontmatter (file default) or a profile (shared base in scout.config.json), device: is an object:

---
feature: Add to Home Screen
device:
  device: iPhone 14          # a Playwright device name (the base)
  userAgent: "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 …)"  # optional override
---

## Sheet renders under iOS Safari
Open the app; the Add-to-Home-Screen sheet appears.

## Same flow on Android
device: Pixel 7              # per-scenario inline override: a device name

Open the app; the native install banner appears instead.
  • Inline override form: the per-## override names a device only — device: <device name>. Individual overrides (userAgent, viewport, …) belong in the frontmatter/profile because a heading override line can’t carry a nested YAML object.
  • Merge: a profile’s device is the base; the file frontmatter and then the per-scenario override win, per field (an inline device: Pixel 7 keeps a file-level userAgent). The viewport field is replaced wholesale (it’s a width/height pair).
  • Composes over the viewport: the resolved device layers on top of the run’s viewport — its viewport/UA/isMobile/hasTouch/deviceScaleFactor win over the viewport’s. The named viewport still determines which sizes the scenario fans out in and the run identity (<slug>@<viewport>).
  • Not secrets: these fields are plain values, so there is no $ENV:VAR resolution (unlike cookies/storage).
  • Fail-fast: an unknown field, an unknown device name, or a bad shape is a hard error at parse — a silently ignored device would produce a misleading verdict.

Flows that open a new tab

When a click opens a new tab or popup (a booking tool, an OAuth window), just describe it — the agent follows into the new tab and scout records the switch as a deterministic step. Console, network, and text assertions always run on the active tab, so a check after the switch verifies the new tab, not the one you came from.

Show/hide toggles (opacity, class, attribute)

Some controls aren’t added and removed from the page — they’re revealed and hidden in place, with opacity, a class swap, or an attribute flip, while staying mounted in the DOM. A plain “is it visible?” text check can’t tell those apart (a browser treats an opacity:0 element that’s still laid out as visible). Just describe the visual state and scout records a structural assertion — a class token (opacity-0 vs opacity-100), an attribute (aria-expanded), or a computed style — that replay re-checks deterministically.

## Menu button toggles the drawer
Tap the menu button; the navigation drawer becomes visible. Tap it again;
the drawer is hidden (it stays in the page, just faded out).

The agent also reaches elements that have no accessible role or name — a gesture/tap layer, an overlay <div> identified only by data-testid — which a role-based click can’t target. You don’t declare selectors; the agent discovers the real element at run time and records a stable data-testid (or CSS fallback) locator.

Asserting console logs & API calls

Beyond what’s on screen, scout can verify the browser console and the network calls the page makes. Just describe the expectation in the prose — the agent observes the real console/network on the first verified run and records a tolerant, deterministic assertion that replay re-checks without an LLM.

## Checkout fires the order API cleanly
The user completes checkout. A POST to /api/checkout returns 2xx and the
response includes an orderId. No errors appear in the browser console.

What the agent records:

  • Network — matched by method + URL pattern (glob with */**) + status class (2xx), optionally requiring stable substrings in the response body (field names like orderId). It deliberately avoids volatile values (ids, timestamps), so the assertion survives replay.
  • Console — “no errors” covers console.error and uncaught exceptions. Known/expected noise can be ignored by substring (e.g. a third-party favicon 404).

You can also assert a specific log appeared — handy for debug output gated behind a flag — not just the absence of errors:

## Debug logging turns on with ?gat-debug=true
Open the page with ?gat-debug=true. A console log containing "DEBUG:[GAT]" appears.

This records a positive console assertion that replay re-checks: it requires a single message containing your substring(s), so match on a stable prefix (DEBUG:[GAT]), never a volatile value — that keeps it tolerant of unrelated console noise.

Keep these expectations about shape, not exact values — “a POST to /api/checkout returned 2xx with an orderId”, not “orderId was ord_42”. Pinning a volatile value is the main way a network assertion turns flaky.

Authoring options

  • By hand — write the markdown directly.
  • CLI — scout create <name> -f <feature> -c <text> [-p profile] [-n notes]. A convenience for humans without an agent.
  • AI agent — your coding agent writes a richer spec straight from repo context. See AI agents & MCP.

Migrating from a legacy scenarios.json

Older scouts kept a single .scout/scenarios.json. Convert it once:

scout migrate   # → one .scout/specs/<slug>.scout.md per scenario, relocates cached scripts, backs up the JSON

It’s idempotent and preserves cached scripts (so replay still works without re-recording). Re-run scout go once afterward to repopulate run status, review the generated feature: frontmatter, and delete .scout/scenarios.json.bak when happy.