Overview

Firegoat runs Agent Experience (AX) evaluations for your developer tool, testing the flows your agentic users typically take. An autonomous agent works through a real task, and you get back a complete evaluation: metrics, regression alerts, and a full session trace.

Everything is scoped to your organization (your email domain). API keys, prompts, runs and reports all belong to one org, and you only ever see your own. Throughout these docs, replace <org> with your org domain (for example acme.com) and $EVALS_API_KEY with a key you create on the Settings page.

New here? Start with the Quickstart.

Ways to interact

There are six entry points, in rough order of how hands-on they are:

• Dashboard: the web UI at firegoat.ai/evals/<org>. Browse prompts, runs and reports, link repos, request evals by hand, and manage keys.
• GitHub App: install once, and every pull request with a preview deployment is evaluated automatically, with the grade posted back as a sticky PR comment.
• GitHub Action: a self-hostable workflow (firegoat-ai/evals-action) you add to your own CI when you'd rather drive runs with an API key than install the App.
• REST API, evals CLI and MCP server: program against the platform directly: list prompts, trigger runs and poll status over REST or from your terminal, or wire the same capabilities into an AI agent over MCP.
• Firegoat CLI: run evals locally with your own claude, no account needed.
• Agent feedback: embed @firegoat/feedback in your CLI or MCP so coding agents can file bugs and requests to your Firegoat feedback page.

Troubleshooting

If a run looks wrong or a request never completes, check the Runs page for its status and open the report's trace to see exactly what the agent did, then confirm your url-map/urlMap actually points at a reachable preview. For the API, the most common mistake is passing template variables that don't match the prompt's declared {{variables}}.