Use Firegoat

Dashboard and settings

The dashboard pages, API keys and integrations.

The dashboard lives at https://firegoat.ai/evals/<org>. Access requires sign-in, and you can only open pages for your own org.

Pages

The left sidebar lists:

  • Dashboard (/<org>): your prompts with their latest grade, score and trend.
  • Prompts (/<org>/prompts): every prompt, and + New Prompt to create one.
  • Runs (/<org>/requests): the status of every eval run: pending, running, completed or failed.
  • PRs (/<org>/pull-requests): open and previously-evaluated PRs from your connected repos. Link repos to prompts and trigger evals by hand here.
  • URLs (/<org>/urls): the URLs, web searches, MCP calls and commands agents used across your runs, with how often and how each went.
  • Providers (/<org>/providers) and Benchmark (/<org>/benchmark): provider and competitor comparisons.
  • Insights (/<org>/insights): recommendations from your eval runs, as a board you can triage.
  • Ask AI (/<org>/chat): a chat assistant over your eval data, backed by the same tools as the MCP server.

Settings (/<org>/settings) and Billing (/<org>/billing) are in the menu under your name, top right.

Settings

The Settings page is where you manage your org and how external systems authenticate:

  • Scheduled evals: the default model for scheduled runs of your prompts.
  • Domains: people signing in with an email at these domains join your org.
  • Credentials: named secrets for your org. Sensitive values are stored write-only and never shown again.
  • Slack Integration: add the Firegoat bot to your workspace (see Slack).
  • GitHub Integration: connect or manage the Firegoat Evals GitHub App (see GitHub pull requests). A connected installation shows its repos and a link to manage them on GitHub.
  • API Keys: create, list and revoke keys. The same key authenticates the REST API, the evals CLI and the MCP server.
  • Bring Firegoat eval data into your own agent: a ready-to-paste command for registering the hosted MCP server in your agent client.

API keys

Give a key a name and click Create key; the full key (prefixed evals_) is shown exactly once. Keys are personal (scoped to you and your org) and can be revoked from the table at any time. Provide the key on every request as the header Authorization: Bearer <key>.