Reference

Evals CLI

Install the evals CLI and call the API from your terminal and CI.

evals talks to the Firegoat evals API from your terminal, scripts and CI.

1. Install

curl -fsSL https://firegoat.ai/evals/install.sh | sh
  • Supports macOS (Apple silicon and Intel) and Linux x64.
  • Installs to ~/.local/bin/evals. If that directory is not on your PATH, the installer prints the line to add to your shell profile: export PATH="$HOME/.local/bin:$PATH".
  • Prefer a manual download? Get the binary and SHA256SUMS from github.com/firegoat-ai/evals-cli/releases.

2. Log in

evals auth login

This opens your browser. Enter the code shown in the terminal and approve. The CLI gets an API key named evals login YYYY-MM-DD for your organization and stores it in a file readable only by you (~/Library/Application Support/evals/auth-keyring.json on macOS, ~/.config/evals/auth-keyring.json on Linux). There is no keychain prompt.

For CI and scripts, skip the browser and export an API key instead. Create one under API keys in your organization's settings (https://firegoat.ai/evals/<your-org>/settings):

export EVALS_TOKEN=<your API key>

An exported EVALS_TOKEN wins over a stored login.

3. Check

evals version
evals auth status
evals whoami

evals version (also evals -v or evals --version) prints the installed version. evals whoami shows the account and organization the CLI calls the API as.

4. Use

evals prompts list
evals prompts get --prompt-id <prompt-id>
evals runs request --prompt-id <prompt-id>
evals runs get --run-id <run-id>
evals runs list
  • --format json|table|yaml|csv picks the output format (table in a terminal, JSON when piped).
  • --query <expr> filters the response with JMESPath, for example --query "[].id".
  • --dry-run validates a request locally without sending it.
  • --help on any command lists its flags; --schema prints them as JSON for agents.
  • evals generate-skills writes SKILL.md files so an AI agent can use the CLI.

5. Log out

evals auth logout --scheme deviceLogin

This removes the login from that file. The API key it created still works until you revoke it under API keys in your organization's settings.

Reference

The full API is described by the OpenAPI spec at firegoat.ai/evals/api/openapi. The REST API page lists the endpoints.