# Writing prompts and checks

> How to write a prompt, end goal and checks that measure your product, with examples.

A prompt measures your product only if the task is real, the agent gets no help a real user's agent would not get, and the checks grade what actually happened. This page is about writing them well. For what each field is, see [Concepts](https://firegoat.ai/docs/concepts#prompts).

## Start from a real task

Take the task from a flow your users already do with an agent: make a first API call, install the SDK, provision a resource, send a message and confirm it arrived, deploy something and open its URL. Your quickstart is a good source.

- **Cover the core features first.** Start with the handful of features your customers pay for, one prompt each. Keep core platform features (a database, auth) apart from secondary ones.
- **One outcome per prompt.** The run's metrics and friction then describe one task, not three.
- **Pick the starting state.** For a core feature, start cold with the `blank` scaffold. `react` and `nextjs` are not cold starts: both ship Barkbook, a small app with no sign-in, which suits an auth product. For a deeper feature, start from an app that already uses your product, built by the setup command. See [Scaffolds](https://firegoat.ai/docs/concepts#scaffolds).

## Write the prompt the way your user would

Write one or two lines in a real user's words, about the problem they want solved.

- **State the goal, not the steps.** "Send a test event and print its id" beats "call `POST /v1/events`". You are measuring whether the agent gets there from your docs, API and errors alone.
- **Name your product, not its internals.** Your user would say "use Acme". They would not name the endpoint, the command, the docs page or the MCP tool. Leaving the product name out measures something else: whether agents pick your product at all.
- **Name the surface when you compare surfaces.** If you run the same task through your CLI, MCP server and SDK, each prompt names its surface. Otherwise each one measures whether the agent found that surface, not how well it works.
- **No hedges or fallbacks.** No "or equivalent", no "if that fails, try the REST API", no "explain why if you can't". A run that stopped at a blocker failed, and the prompt should let it fail.
- **Tell the agent what a user would.** "My API key is in `ACME_API_KEY`" is fine: a user would say it. A hint at the answer is not.

## Write the end goal

The judge grades the end goal first on every run. Make it one concrete outcome that either happened or did not, with no escape clauses and no hints.

- **Make the result checkable.** When the answer can be computed in advance, end the prompt with "Save to /app/result.json with exactly this shape: ..." so the result can be compared field by field.
- **Use unique, harmless objects.** Put a timestamp in every name the agent creates, use `example.com` addresses and test data, and end with "Only create these objects; do not change or delete anything else."

## Write the checks

Checks are where your product knowledge goes. The prompt stays a plain user request; the checks say what a correct result is and how the agent was supposed to use your product: the right command, MCP tool, endpoint or SDK call.

A check is a slug and one sentence the judge reads, for example `"webhook-received": "The app logged a webhook event from Acme"`.

- **One observable outcome per check.** No "and", "or" or "or equivalent", and no partial credit. Split a two-part check into two.
- **No check passes for reporting a blocker.** "Set up the webhook or explained why it couldn't" passes a run that did nothing.
- **Grade outcomes, not form.** A check that asks for a file name, a variable name or exact code the task never specified fails good runs. Ask for the behavior instead.
- **Keep the goal next to the method.** A check that restates the outcome is fine alongside checks on how your product was used. A run that reached the goal the wrong way, or used your product right and missed the goal, then shows which.
- **Turn off a default check only when it cannot apply,** for example `build` on a task that writes no code. See [Checks](https://firegoat.ai/docs/concepts#checks) for the defaults.

Over the API and MCP, `checks` is a map of slug to sentence that merges on top of the defaults. A sentence adds a check or replaces the default with the same slug, and `null` turns a default off. Slugs are lowercase letters, digits, `-` and `_`. Check text is not secret: it reads back verbatim from `get_prompt`. An edit applies to evals requested after it.

### Judged checks and script checks

Every check on a cloud prompt is judged: the judge reads the run and cites the turns of the trace it used as evidence. Write checks about things that show up there: a command and its output, a tool call, a response.

A judge can be fooled by a confident wrong answer, such as a stale version number reported as current. When the result matters, compare it with the real state yourself: read the rows, the deployed URL or the stored object back with your own read-only credentials, at the time of the run. Files the agent wrote are in the run's code archive (`GET /api/v1/runs/{runId}/artifacts`).

Script checks, shell commands whose exit code passes or fails a test, are a feature of the [Firegoat CLI](https://firegoat.ai/docs/firegoat-cli), which runs evals on your own machine. Use them there for anything a command can verify exactly.

## Logins, credentials and the browser agent

Decide what the agent should already have when it starts, then set that up.

- **To measure the product once the agent is in,** give it the credential as an [environment variable](https://firegoat.ai/docs/concepts#environment-variables-and-logged-in-clis). A CLI that reads its token from the environment is logged in from the first command. Over the API or MCP, `envCredentials` sets a variable from a saved [credential](https://firegoat.ai/go/settings/credentials), so editing the credential reaches the next run.
- **To measure sign-up and getting a key,** leave the credentials out and write **Persona instructions** for the browser agent: what the simulated user knows and will hand over when the coding agent stops and asks. The coding agent never reads them, so instructions meant for the agent do nothing there. Only Claude Code runs with the browser agent; see [Supported agents and models](https://firegoat.ai/docs/agents-and-models#the-browser-agent).
- **Expect `no_creds` when the agent never authenticates.** A run with no sign of working credentials has no score.

The setup command runs before the agent's environment exists, so it cannot read the prompt's variables. Use it to install the version of your CLI, SDK or MCP server under test. Keep commands that need a token in the prompt, and end setup with a command that fails when the install did not happen, so a broken setup shows as a setup failure, not an agent failure.

## Scenarios

A [scenario](https://firegoat.ai/docs/concepts#scenarios) runs the same task from a different starting point. Use one when the question is "same goal, different conditions": logged in versus cold, an empty directory versus an existing app, your CLI versus your MCP server. A scenario inherits its parent's checks unless you customize them for it, so the runs stay comparable.

## Before you compare results

Request one run first to prove the setup works, then raise **Runs per eval** to three or more before you compare anything. Use [template variables](https://firegoat.ai/docs/concepts#prompts) for values that change per run, such as a region or a resource name, never to pass the answer.

## Common mistakes

| Mistake | Fix |
| --- | --- |
| The prompt lists the steps, commands or docs URL. | State the goal. Put the expected method in checks. |
| The end goal or a check has a fallback ("or equivalent", "or explain why"). | One outcome that happened or did not. |
| A check combines two outcomes. | Split it into two checks. |
| A check asks for an exact file name or code the task never specified. | Check the behavior. |
| An MCP eval has no check that the work went through the MCP server. | Add a check that the work came from your MCP tool calls, not `curl`, the SDK or the REST API. |
| The token is described in persona instructions. | Set it as an environment variable. |
| The setup command reads an environment variable. | Move that command into the prompt. |
| The setup command clones its own app into the `react` or `nextjs` scaffold. | Pick `blank`, or the app lands on top of Barkbook. |
| One run is read as the result. | Request several and read them together. |

## Examples

These use Acme, a fictional events API.

### A prompt that hands over the answer

Weak:

```text
Prompt: Go to docs.acme.dev/quickstart, run npm install @acme/node, create a
client with new Acme(process.env.ACME_API_KEY) and call client.events.send()
with type "test". If the SDK doesn't work, use the REST API instead.
End goal: The agent tried to send an event.
```

The prompt names the page, the package and the call, so the run measures whether the agent can follow instructions. The fallback lets it skip the SDK, and the end goal passes a run that only tried.

Strong:

```text
Prompt: Send a test event to Acme from this Node project and print the event id
Acme returns. My API key is in ACME_API_KEY.
End goal: A script in /app sends one event to Acme and prints the event id that
Acme returned.
```

```json
{
  "used-sdk": "The script sends the event with the official Acme Node SDK.",
  "id-printed": "The script's output in the trace shows an event id returned by Acme.",
  "key-from-env": "The script reads the API key from ACME_API_KEY."
}
```

The method the docs should lead to (the SDK) lives in a check, and each check is one outcome.

### A check that is too brittle, and one that is too lenient

```json
{
  "client-file": "The agent created src/lib/acme.ts containing exactly new Acme({ apiKey: process.env.ACME_API_KEY }).",
  "webhook": "The agent set up a webhook handler or explained why it couldn't."
}
```

The first fails a correct run that named the file differently or formatted the call another way. The second passes a run that gave up. Rewritten:

```json
{
  "client-from-env": "The Acme client reads its API key from the ACME_API_KEY environment variable.",
  "webhook-received": "The app logged a webhook event from Acme."
}
```

### An MCP server eval

```text
Prompt: Using the Acme MCP server, create a dataset named
eval-<current UTC time as YYYYMMDDHHMM> with 5 rows, where row i has country
["US","DE","JP","BR"][i mod 4], then read it back. Save to /app/result.json
with exactly this shape: {"dataset": string, "read_back": [{"country": string}]}.
Only create these objects; do not change or delete anything else.
```

```json
{
  "created-via-mcp": "The dataset and its rows were created through Acme MCP tool calls in the trace.",
  "readback-matches": "/app/result.json read_back equals an Acme MCP read in the trace that returned 5 rows.",
  "stayed-in-mcp": "The work came from Acme MCP tool calls, not from curl, the Acme SDK, CLI or REST API, or another MCP server."
}
```

The prompt names the server but no tool. The seed is computed from the row number, so the right answer is known before the run. `stayed-in-mcp` keeps a run that gave up on the server and called the API directly from passing.

<Tip>
To have your own coding agent draft prompts with this guidance, install the Firegoat skills: `npx skills add firegoat-ai/skills`.
</Tip>
