Local evals (Firegoat CLI)

The Firegoat CLI runs AX evals on your own machine: no account, no server. Run it in the repo of the tool you want to test. It needs Node 18+ and claude installed and logged in. The product page is firegoat.ai/cli.

Install and run

1. Init. Creates firegoat.yaml and .firegoat/, installs the Firegoat skills, and copies a prompt for your coding agent.
2. Write tests. Edit firegoat.yaml yourself, or paste the copied prompt into your coding agent and let it write them.
3. Eval. Your claude runs each test in its own fresh folder. Checks run after, and set the exit code.

npx firegoat init     # firegoat.yaml + .firegoat/, skills, agent prompt
npx firegoat eval     # runs claude on each test in .firegoat/runs/<runId>/<test>/<n>/work

Full yaml reference: npx firegoat init --help. All eval flags: npx firegoat eval --help.

firegoat.yaml

Each test is a prompt, an end_goal, and script checks. build builds your code once, so the agent uses your local build, not the registry. install installs that build into each fresh folder.

build: npm pack --pack-destination "$FIREGOAT_OUT"
install: npm init -y >/dev/null && npm install "$FIREGOAT_OUT"/*.tgz
tests:
  - name: quickstart
    prompt: >-
      acme-sdk is already installed in this project. Use it to send one test event
      and print the event id. Credentials are in ACME_API_KEY. Keep it minimal.
    end_goal: A script sends one event with the SDK's client and prints its id.
    checks:
      - run: grep -qE 'evt_[A-Za-z0-9]+' "$FIREGOAT_TRANSCRIPT"

For a CLI, whatever build writes to $FIREGOAT_OUT is first on the agent's PATH. runs: 3 repeats every test, and {{vars.SHA}} in a prompt is filled from --var on the command line.

npx firegoat eval --json                      # all tests, JSON summary on stdout
npx firegoat eval quickstart --runs 3         # one test, three runs
npx firegoat eval --var SHA=$(git rev-parse HEAD)
npx firegoat eval --parallel 1                # one run at a time (default 4)

How it works

1. Runs locally. Every run gets a fresh, empty folder: .firegoat/runs/<runId>/<test>/<n>/work. Four runs at a time by default.
2. Your own claude. It runs on your Claude login, or ANTHROPIC_API_KEY if set. Its config dir is fresh: none of your settings, CLAUDE.md, plugins, skills or MCP servers. The agent starts cold, like your users' agents.
3. Traces stay on disk. Each run saves transcript.txt (every tool call, its output, the final answer, files made), run.json, and claude's full session. summary.json covers the whole eval.
4. Checks set the exit code. Script checks run in the agent's folder after it finishes.
5. Your coding agent judges the end goal. Firegoat records no verdict beyond the checks. Your coding agent runs npx firegoat eval --json, reads each run's run.json and transcript, and decides whether the end_goal was met.

Feedback

Something got in your way? Tell us from the terminal: npx firegoat feedback "..."