Admin console · animated

Click to expand

Scheduled tests

Route: /qa-coverage (sidebar: Checks › Scheduled tests; formerly QA Coverage)

Scenario: You shipped a login flow last week and want to know automatically if it still works — without writing a full Playwright suite from scratch.

Scheduled tests in Mushi Mushi let you describe user flows in plain English, run them on a schedule, and get screenshots plus failure evidence when something breaks.

Story lifecycle

Create story (NL prompt or Playwright script)
  → stored in qa_stories (schedule_cron: '0 * * * *' default)
  → pg_cron fires qa-story-runner every minute
  → BYOK key resolution (firecrawl / browserbase / openai)
  → execute via provider
  → write qa_story_runs (status, latency_ms, provider_session_url, assertion_failures)
  → write qa_story_evidence (screenshots, console logs, HAR)
  → on fail/recover → Slack Block Kit (transition-aware) + Discord webhook + dispatchPluginEvent

Execution providers

To use Browserbase or Firecrawl, add your key in Settings → Web tools first, then select the provider when creating a story.

Creating a story

1. Click + New story (header or empty state).
2. In the New QA story modal, fill in:
   • Name — short label for the story card
   • Target URL — the page the runner opens first (required for Firecrawl and Browserbase)
   • Prompt — describe what to verify — your user story in plain English (e.g. “Open the Reports page, click the first report, verify the classification badge is visible”)
   • Browser provider — Firecrawl Actions (recommended) / Browserbase / Local Playwright (CLI only)
   • Run schedule — every hour (default), every 6 hours, daily, weekly, or a custom cron
3. Click Create story.

In the story drawer you can change the target URL and turn the story on or off.

Story cards

The main grid shows one card per story with:

• Provider badge — Local / Browserbase / Firecrawl
• 24h pass rate — colour-coded bar and percentage
• 7 days — one square per UTC day: red when a run failed that day, green when every finished run passed, grey with none. Skipped runs never count as failures. From last_7d on GET /v1/admin/projects/:id/qa-coverage (null when the runs could not be read).
• Run count and last run time
• disabled / queued chip when applicable
• Link to last failure Browserbase replay session

Story drawer

Click any card to open the story drawer:

• Story prompt and provider at the top
• Script section (collapsible) — the raw Playwright script if one was generated
• Run history — newest runs first. Expand any run to see:
  • Meta (status, provider, latency)
  • Summary and error message
  • Assertion failures table — each failing assertion listed
  • Evidence — screenshots, video thumbnail, HAR download
  • Browserbase replay link (if Browserbase was used)
• A live indicator and 3-second auto-poll when a run is active

Manual run

Click Run now on any story to trigger an immediate execution regardless of its schedule. The story must be on: a story that is off says “Turn the story on first”; turn it on in the story drawer.

Filters

The chips above the grid filter the stories: All, Failing, Passing, No runs in 24h and Turned off. The 24-hour pass rates come from the qa_story_coverage_24h materialised view, refreshed every 15 minutes.

Related pages

• Mushi Mushi health — verify Firecrawl / Browserbase keys
• Improvement runs — producer and judge iteration loop
• Fixes — act on test failures