Local Agent Experience (AX) evals

Watch an agent use your SDK, CLI, API and docs.

See exactly where agents get stuck using your product, and fix it before they give up.

npx firegoat init

Then paste the prompt it prints into your agent and let it set up the evals.

npx firegoat eval

Node 18+. claude installed and logged in.

01 Install

Your first eval runs in minutes.

No account, no server. Run it in the repo of the tool you want to test.

1
Init
Creates firegoat.yaml and .firegoat/, installs the Firegoat skills, and copies a prompt for your coding agent.
2
Write tests
Edit firegoat.yaml yourself, or paste the copied prompt into your coding agent and let it write them.
3
Eval
Your claude runs each test in its own fresh folder. Checks run after, and set the exit code.

$ npx firegoat init     # firegoat.yaml + .firegoat/, skills, agent prompt
$ npx firegoat eval     # runs claude on each test in .firegoat/runs/<runId>/<test>/<n>/work

Full yaml reference: npx firegoat init --help. All eval flags: npx firegoat eval --help.

02 Examples

Describe your tests in firegoat.yaml.

Each test is a prompt, an end_goal, and script checks. build builds your code once, so the agent uses your local build, not the registry.

npm package

Can an agent get started with my npm package?

build packs your package once. install installs that build into each fresh folder. The check greps the transcript for a real event id.

build: npm pack --pack-destination "$FIREGOAT_OUT"
install: npm init -y >/dev/null && npm install "$FIREGOAT_OUT"/*.tgz
tests:
  - name: quickstart
    prompt: >-
      acme-sdk is already installed in this project. Use it to send one test event
      and print the event id. Credentials are in ACME_API_KEY. Keep it minimal.
    end_goal: A script sends one event with the SDK's client and prints its id.
    checks:
      - run: grep -qE 'evt_[A-Za-z0-9]+' "$FIREGOAT_TRANSCRIPT"

CLI

Does my CLI make sense to an agent?

Whatever build writes to $FIREGOAT_OUT is first on the agent's PATH. The check proves the agent found acme deploy.

build: go build -o "$FIREGOAT_OUT/acme" ./cmd/acme
tests:
  - name: deploy
    prompt: Deploy the hello-world template to a preview environment with the acme CLI and print the URL.
    end_goal: The agent finds the right acme commands and ends with a working preview URL.
    checks:
      - run: grep -q '\$ acme deploy' "$FIREGOAT_TRANSCRIPT"

Python, 3 runs, vars

Same test, three runs, pinned to a commit.

runs: 3 repeats every test. {{vars.SHA}} is filled from --var on the command line.

runs: 3
build: uv build --wheel --out-dir "$FIREGOAT_OUT"
install: uv venv && uv pip install "$FIREGOAT_OUT"/*.whl
tests:
  - name: sandbox-hello
    prompt: >-
      Using the acme Python SDK (already installed), create a sandbox on commit
      {{vars.SHA}}, run `echo hello-from-acme` in it, print the output, then delete it.
    end_goal: The script prints hello-from-acme from a sandbox and deletes the sandbox.
    checks:
      - run: grep -rq hello-from-acme "$FIREGOAT_WORKDIR"

$ npx firegoat eval --json                      # all tests, JSON summary on stdout
$ npx firegoat eval quickstart --runs 3         # one test, three runs
$ npx firegoat eval --var SHA=$(git rev-parse HEAD)
$ npx firegoat eval --parallel 1                # one run at a time (default 4)

03 How it works

Your machine. Your agent. Your traces.

Firegoat is a thin wrapper. It makes folders, runs your claude, saves what happened, and runs your checks.

1Runs locally

Every run gets a fresh, empty folder: .firegoat/runs/<runId>/<test>/<n>/work. Four runs at a time by default.

2Your own claude

It runs on your Claude login, or ANTHROPIC_API_KEY if set. Its config dir is fresh: none of your settings, CLAUDE.md, plugins, skills or MCP servers. The agent starts cold, like your users' agents.

3Traces stay on disk

Each run saves transcript.txt (every tool call, its output, the final answer, files made), run.json, and claude's full session. summary.json covers the whole eval.

4Checks set the exit code

Script checks run in the agent's folder after it finishes.

0	every run's checks passed
1	a check failed
2	a run was not graded (claude missing or logged out, crash, timeout, build or install failed)
3	usage or config error

5Your coding agent judges the end goal

Firegoat records no verdict beyond the checks. Your coding agent runs npx firegoat eval --json, reads each run's run.json and transcript, and decides whether the end_goal was met. Then it fixes your docs.

Something got in your way?

Tell us from the terminal: npx firegoat feedback "..."

Firegoat on npm Agent Arena