Get started

Quickstart

Sign in, create a prompt and read your first eval report.

Firegoat runs AX evals for you: an AI agent does a task with your product, the same way one of your users might, and you get a report back. This page walks you through requesting an account and running your first eval.

If you'd rather try it without an account, the Firegoat CLI can run evals locally using your own claude.

1. Get access

Firegoat is organized around your work email: when you sign up, you'll only see prompts, runs and reports for your own organization.

Go to https://firegoat.ai/request-access and request an account. Once you're approved, sign in at https://firegoat.ai/evals with your work email. You'll land on your organization's dashboard.

2. Create a prompt

A prompt is the task you want the agent to do.

In the sidebar, click Prompts, then + New Prompt. On the Basics step, fill in:

  • Name
  • Prompt for the coding agent: the task itself, phrased the way your user would ask an agent to do it
  • End goal: what success looks like. This is optional, but if you set it we'll grade whether the agent reached it

The other steps (Browser agent, Scenarios, Checks, Run setup) are all optional.

Every step has a Create and run button. Clicking it saves the prompt and requests its first eval.

Here's an example:

Prompt: "Add Acme authentication to a new Next.js app. Users should be able
to sign up and sign in with email."
End goal: "A user can sign up and sign in through Acme in the running app."

3. Watch the run

Open Runs in the sidebar. It lists every run with its status: pending, running, completed or failed.

4. Read the report

Open the finished run's report. It has a score and a grade, a pass/fail checklist (the end goal first, then default checks such as "Relevant packages/dependencies were installed" and "The code successfully called the provider's API"), metrics for time, cost, errors and interruptions, recommendations, and the full trace of every tool call the agent made.

Start with the checklist, then open the trace to see where the agent got stuck.

5. Run it from your terminal

Install the evals CLI and log in. evals auth login opens your browser: enter the code shown in the terminal and approve.

curl -fsSL https://firegoat.ai/evals/install.sh | sh
evals auth login

Find your prompt's id, request a run, then check on it. evals runs get shows the run's status and, once it finishes, its report.

evals prompts list
evals runs request --prompt-id <prompt-id>
evals runs get --run-id <run-id>

Next steps