<!-- firegoat.ai/cli.md, and firegoat.ai/cli for `Accept: text/markdown` (vercel.json). The markdown twin of index.html: edit the two together, same copy. -->

# Firegoat CLI

Local Agent Experience (AX) evals.

## Watch an agent use *your* SDK, CLI, API and docs.

See exactly where agents get stuck using your product, and fix it before they give up.

```sh
npx firegoat init
```

Then paste the prompt it prints into your agent and let it set up the evals.

```sh
npx firegoat eval
```

Node 18+. `claude` installed and logged in.

## 01 Install

### Your first eval runs in minutes.

No account, no server. Run it in the repo of the tool you want to test.

1. **Init.** Creates `firegoat.yaml` and `.firegoat/`, installs the Firegoat skills, and copies a prompt for your coding agent.
2. **Write tests.** Edit `firegoat.yaml` yourself, or paste the copied prompt into your coding agent and let it write them.
3. **Eval.** Your `claude` runs each test in its own fresh folder. Checks run after, and set the exit code.

```sh
npx firegoat init     # firegoat.yaml + .firegoat/, skills, agent prompt
npx firegoat eval     # runs claude on each test in .firegoat/runs/<runId>/<test>/<n>/work
```

Full yaml reference: `npx firegoat init --help`. All eval flags: `npx firegoat eval --help`.

## 02 Examples

### Describe your tests in firegoat.yaml.

Each test is a prompt, an `end_goal`, and script checks. `build` builds your code once, so the agent uses your local build, not the registry.

#### npm package: Can an agent get started with my npm package?

`build` packs your package once. `install` installs that build into each fresh folder. The check greps the transcript for a real event id.

```yaml
build: npm pack --pack-destination "$FIREGOAT_OUT"
install: npm init -y >/dev/null && npm install "$FIREGOAT_OUT"/*.tgz
tests:
  - name: quickstart
    prompt: >-
      acme-sdk is already installed in this project. Use it to send one test event
      and print the event id. Credentials are in ACME_API_KEY. Keep it minimal.
    end_goal: A script sends one event with the SDK's client and prints its id.
    checks:
      - run: grep -qE 'evt_[A-Za-z0-9]+' "$FIREGOAT_TRANSCRIPT"
```

#### CLI: Does my CLI make sense to an agent?

Whatever `build` writes to `$FIREGOAT_OUT` is first on the agent's `PATH`. The check proves the agent found `acme deploy`.

```yaml
build: go build -o "$FIREGOAT_OUT/acme" ./cmd/acme
tests:
  - name: deploy
    prompt: Deploy the hello-world template to a preview environment with the acme CLI and print the URL.
    end_goal: The agent finds the right acme commands and ends with a working preview URL.
    checks:
      - run: grep -q '\$ acme deploy' "$FIREGOAT_TRANSCRIPT"
```

#### Python, 3 runs, vars: Same test, three runs, pinned to a commit.

`runs: 3` repeats every test. `{{vars.SHA}}` is filled from `--var` on the command line.

```yaml
runs: 3
build: uv build --wheel --out-dir "$FIREGOAT_OUT"
install: uv venv && uv pip install "$FIREGOAT_OUT"/*.whl
tests:
  - name: sandbox-hello
    prompt: >-
      Using the acme Python SDK (already installed), create a sandbox on commit
      {{vars.SHA}}, run `echo hello-from-acme` in it, print the output, then delete it.
    end_goal: The script prints hello-from-acme from a sandbox and deletes the sandbox.
    checks:
      - run: grep -rq hello-from-acme "$FIREGOAT_WORKDIR"
```

```sh
npx firegoat eval --json                      # all tests, JSON summary on stdout
npx firegoat eval quickstart --runs 3         # one test, three runs
npx firegoat eval --var SHA=$(git rev-parse HEAD)
npx firegoat eval --parallel 1                # one run at a time (default 4)
```

## 03 How it works

### Your machine. Your agent. Your traces.

Firegoat is a thin wrapper. It makes folders, runs your `claude`, saves what happened, and runs your checks.

1. **Runs locally.** Every run gets a fresh, empty folder: `.firegoat/runs/<runId>/<test>/<n>/work`. Four runs at a time by default.
2. **Your own claude.** It runs on your Claude login, or `ANTHROPIC_API_KEY` if set. Its config dir is fresh: none of your settings, CLAUDE.md, plugins, skills or MCP servers. The agent starts cold, like your users' agents.
3. **Traces stay on disk.** Each run saves `transcript.txt` (every tool call, its output, the final answer, files made), `run.json`, and claude's full session. `summary.json` covers the whole eval.
4. **Checks set the exit code.** Script checks run in the agent's folder after it finishes.

   | Exit code | Meaning |
   | --- | --- |
   | 0 | every run's checks passed |
   | 1 | a check failed |
   | 2 | a run was not graded (claude missing or logged out, crash, timeout, build or install failed) |
   | 3 | usage or config error |

5. **Your coding agent judges the end goal.** Firegoat records no verdict beyond the checks. Your coding agent runs `npx firegoat eval --json`, reads each run's `run.json` and transcript, and decides whether the `end_goal` was met. Then it fixes your docs.

## Something got in your way?

Tell us from the terminal: `npx firegoat feedback "..."`

- [Firegoat on npm](https://www.npmjs.com/package/firegoat)
- [Agent Arena](https://firegoat.ai/arena)
- [Request access](https://firegoat.ai/request-access)