Concepts

An eval is a coding agent working through one task with your product in a fresh sandbox, then a judge grading what happened. These are the pieces you work with.

Prompts

A prompt is the task definition: what the agent is asked to do, and what success looks like. You create one from + New Prompt on the Prompts page, with POST /api/v1/prompts, or with the create_prompt MCP tool. Creating a prompt requests its first eval.

Field	What it is
Name	The title shown in the dashboard.
Prompt for the coding agent	The task, written the way your user would ask their agent.
End goal	What success looks like. The judge grades it first on every run.
Environment variables	Set on the coding agent's own process, for example an API token. Values are write-only.
Setup command	A shell script that prepares the sandbox before the agent starts, such as npm install.
Scaffold	The starter codebase the sandbox is built from, for example an empty directory or a demo app.
Persona instructions	What the simulated user knows and hands over when the agent stops and asks.
Checks	Extra pass/fail criteria for the judge. See Checks.
How many runs per eval, How often	The cadence: daily, weekly, bi-weekly, monthly, or none (never re-run automatically). A prompt with no cadence set runs weekly.

A prompt is active (runs on its cadence), suggested (authored but not on a cadence yet, shown as One-off prompts) or archived (off the schedule, but you can still run it by hand). Once a published report is linked to a prompt, its text is locked so past and future runs stay comparable; the title and end goal stay editable.

Template variables. Write {{name}} in the prompt text or setup command and supply the value on each run (templateArgs over the API, template-vars in the GitHub Action). A run without them is rejected.

Scenarios

A scenario is a variant of a prompt: the same product, a different starting point. It has its own name and can override the prompt text, setup command, persona instructions and environment. A field left unset inherits the parent prompt's value; an explicitly empty one turns it off. Scenarios follow the parent's schedule and cannot have template variables. Add them on the prompt's Scenarios tab or the Scenarios step of + New Prompt.

Competitors

Add a competitor to a prompt by its URL and Firegoat runs the same task against that product as a separate competitor lane. The Benchmark page compares your results with theirs. Removing a competitor archives the lane and keeps its run history.

Checks

Every run is graded on a checklist. The end goal is always checked first, then the default checks:

Check	Passes when
installed	Relevant packages/dependencies were installed
build	Code runs without syntax errors
data	Real data is produced, not placeholder or dummy data
schema	Proper SDK/client configuration exists
client	The provider's SDK or API is actually used in the code
not_placeholder	No TODO comments or placeholder API keys
not_hardcoded	Credentials are externalized, not hardcoded in source
api_reachable	The code successfully called the provider's API

On a prompt's Checks tab you can turn defaults off and add your own. A custom check is a slug and a sentence the judge reads, for example "webhook-received": "The app logged a webhook event from Acme". Write each one as a single observable outcome that either happened or did not. Edits apply to future runs; past runs keep the results they were graded with.

Runs

A run is one attempt at a prompt. Its status moves from pending to running to completed or failed. A completed run also has an outcome: succeeded, or goal_not_met when the agent finished without reaching the end goal. You can cancel a run that has not started yet.

Each run gets a fresh sandbox built from the prompt's scaffold. The setup command runs first, then the coding agent starts with the prompt text and its environment variables. A run that fails during setup has no report; its trace shows the setup command and its output instead.

Runs come from the prompt's cadence, from Request eval in the dashboard, from the API or evals CLI, and from pull requests.

Reports

A finished run publishes a report:

• a score and a grade,
• the checklist, with the turns of the trace each check cites as evidence,
• metrics: time, cost, errors and interruptions,
• recommendations and friction points,
• the full trace: every message and tool call the agent made, plus the browser agent's screenshots when there are any.

The dashboard's prompt page shows trends across runs, so judge a change by several runs, not one.

Insights

Insights are recommendations from your eval runs, collected on a board where you set each card's status (open, accepted, dismissed or fixed), assign it and comment on it. You can also add cards by hand.

Environment variables and logged-in CLIs

To test what an agent does once it is already authenticated, give the prompt the token as an environment variable instead of describing a login. Add it under Environment variables when you create the prompt (or in the prompt's settings), for example GITHUB_TOKEN. Every eval request snapshots the prompt's variables and sets them in the coding agent's shell, so any CLI that reads a token from the environment is logged in from the first command: gh auth status reports the account as logged in through GITHUB_TOKEN, with no browser login.

• Values are write-only. After saving you only see that a key is set, never the value.
• The setup command does not see them. It runs before the agent starts, so keep commands that need the token (gh repo clone, gh pr create) in the prompt for the agent, and use setup only for installs such as npm install.
• Persona instructions are not needed for this. They go to the simulated user, who only hands them over when the agent stops and asks. The agent never reads them, so "use the GITHUB_TOKEN env var" there changes nothing; leave them empty or describe the user.