Skip to Content
v1.33.6 · shipped One-screen report · Looks like your app Read the changelog →
Admin consoleAI prompts
A/B traffic split between active and candidate prompts · · open live demo ↗

AI prompts

Route: /prompt-lab (sidebar: AI prompts; formerly Prompt lab). /fine-tuning redirects here.

AI prompts is a Lab: it is under More pages in the sidebar, and Settings → Display & sidebar can keep it in the sidebar. The page works by URL either way.

First on the page: Versions and their grades, one row per AI step (Fast filter, Diagnosis, Grading, Fix writer, …) with the version in use, the best version being tested, their grades, and a verdict: ahead (consider promoting), behind, or waiting for grades. A row opens that step’s editing table.

Scenario: Your fix-worker has been generating patches that touch the wrong file 20% of the time. You suspect the Stage 1 classifier is mislabelling the component field — so the fix agent is looking at the wrong area of the codebase. You want to try a new prompt, test it on real traffic, and promote it only if the judge confirms it’s better than what’s live.

AI prompts is where you iterate on the prompts behind each Mushi Mushi AI step. You do not redeploy anything: a version takes effect on the share of traffic you give it.

The 4-step workflow

The How prompt tuning works panel lists the steps:

1. Build eval dataset → 2. Clone a baseline → 3. A/B at low traffic → 4. Promote when judge ≥ 80%

Step 1 — Build eval dataset: grades need classified reports. The Eval dataset card shows the recent classified reports the grader can use.

Step 2 — Clone a baseline: click Clone (the copy icon) on the version in use. The copy starts at 0% traffic, so it serves no live requests yet. Click Edit to change its template. Steps marked Built-in cannot be cloned: no worker reads a project copy of that prompt yet.

Step 3 — A/B at low traffic: use Set A/B traffic share to give the copy a small share (for example 10%). The judge runs nightly and grades both versions. Compare their Judge score columns once each has a fair number of Evals.

Step 4 — Promote: click Make this the live prompt and confirm Activate at 100%. This removes all A/B splits. The nightly judge run also promotes a candidate on its own once it has at least 30 evaluations and scores more than 5% above the version in use.

The judge runs once per night, so an A/B test needs a few nights of data. Don’t promote after a single night.

Summary tiles

TileWhat to watch
In usePrompt versions serving traffic
Being testedCandidate versions in an A/B test
Best gradeThe best judge score among the versions

Steps

Each tab is one AI step. Only steps with prompt rows appear. Start with the one causing your problem:

StepPrompt controls
Fast filter (stage1)Spam detection: edit if legitimate reports are being dropped
Diagnosis (stage2)Category, severity and component: edit if labels are wrong
Grading (judge)The grader: edit if its scores don’t match your own judgement
Fix writer (fix)Code patch generation
Weekly summary (intelligence)The weekly insights digest
Question to SQL / Answer summaryThe SQL query page
Test reports (synthetic)Generated test reports

Other steps (for example Dependency upgrades, Prompt tuner, Story map audit) show up when they have prompt rows.

Prompt table columns

ColumnWhat it tells you
VersionThe version name, with an Auto badge when the weekly prompt tuner drafted it
StateActive, Candidate or Idle, plus Global for a built-in version
TrafficWhat share of live requests this version serves
Judge scoreHigher is better. Uses the rolling 7-day evaluation window
EvalsHow many evaluations the score is based on. A low count means an unreliable score
UpdatedWhen the version last changed

Actions per row

ActionWhen to use it
CloneStarting point for any edit. Built-in versions are read-only; clone first
EditOpen the template editor for a project version
DiffCompare against the version it was cloned from. Use before promoting
Set A/B traffic shareRaise a candidate’s traffic share gradually
Make this the live promptActivate at 100% and remove all A/B splits
DeleteRemove a candidate that didn’t work out. Its past evaluations stay

Common tasks

Fixing wrong component classification

  1. Open the Diagnosis tab.
  2. Clone the version in use.
  3. Edit the template: add explicit examples of the component names your codebase uses.
  4. Give it a 10% traffic share.
  5. Wait a few nights. Is its Judge score above the version in use?
  6. If yes: Diff, then make it the live prompt. If not: edit and try again.

Investigating a bad judge score

  1. Open the tab for the step with the low score.
  2. Check Evals. With fewer than 10, the score isn’t meaningful yet: wait.
  3. With more than 20 and a still-low score, the prompt is likely wrong. Clone and start fresh.

Advanced: fine-tuning & synthetic

This closed panel at the foot of the page holds two rarely used tools:

  • Fine-tuning jobs: export classified reports and train a small model. See Fine-tuning.
  • Synthetic reports: generate artificial test reports that match your real report patterns, to grow the set of reports the grader can score.

API

GET    /v1/admin/prompt-lab
POST   /v1/admin/prompt-lab/prompts        { stage, name, promptTemplate }
PATCH  /v1/admin/prompt-lab/prompts/:id    { isActive, trafficPercentage, promptTemplate }
DELETE /v1/admin/prompt-lab/prompts/:id
Last updated on
LessonsFine-tuning