Reference

MCP server

Give an AI agent access to your eval data over MCP.

Firegoat speaks the Model Context Protocol, so you can give an AI agent (Claude Code, Claude Desktop, and similar clients) direct read-and-write access to your eval data. The server is a stateless Streamable-HTTP endpoint at https://firegoat.ai/evals/api/mcp, authenticated with the same Authorization: Bearer <api-key> header. Every tool is org-scoped to your key.

Install

Create a key on the Settings page and run the command shown there:

npx mcp-add -n firegoat -t http -u https://firegoat.ai/evals/api/mcp -H "Authorization=Bearer <your-api-key>"

Claude Desktop only loads local command servers, so install it through the mcp-remote bridge, then restart Claude Desktop:

npx install-mcp@1.10.2 https://firegoat.ai/evals/api/mcp --client claude --name firegoat --header "Authorization: Bearer <your-api-key>" --oauth no -y

Or add it to a client config directly. Claude Code expands ${EVALS_API_KEY} from your environment, so the key stays out of a .mcp.json you may commit:

.mcp.json
{
  "mcpServers": {
    "firegoat": {
      "type": "http",
      "url": "https://firegoat.ai/evals/api/mcp",
      "headers": { "Authorization": "Bearer ${EVALS_API_KEY}" }
    }
  }
}

Tools

Tools. The server exposes 48 tools:

ToolWhat it does
list_promptsList your organization's eval prompts (the task definitions agents run). by default only ACTIVE prompts are returned: pass status: 'suggested' for prompts authored but not yet promoted to a cadence (what the dashboard calls One-off prompts, including the ones create_prompt wrote with status: 'suggested'), or status: 'archived' / includeArchived='1' (or 'true') for archived ones — an empty active list does not mean the org has no prompts. competitor sub-prompts are excluded by default; pass includeSubPrompts='1' to include them with parentPromptId and targetProviderId. use get_prompt for one prompt's config, or get_prompt_benchmarks for its run history + competitor comparison.
get_promptGet a single prompt's full configuration (text, end goal, frequency, template vars, loginHintsStatus: set | inherited | none, linkedRepos: the GitHub repos linked with link_repo) by id.
get_prompt_benchmarksGet a prompt plus its run history AND competitor benchmarking (grouped per competitor provider with their own run history). mirrors the prompt detail page.
get_prompt_statsGet rollup stats for a batch of prompts (the prompt-card metrics): current vs previous run-average time/tokens/errors/interruptions with run counts, the models each prompt ran on, its competitor rank, and its author. pass the prompt ids to fetch; prompts you can't see are omitted.
get_prompt_period_comparisonGet the GROUP-WEEKLY aggregate for a parent prompt: self plus every competitor lane of this prompt, summed and averaged over the given window vs the prior same-length window. Returns runCount, avgCostCents, avgDurationMs, totalErrors, totalInterruptions and their deltas for both windows, plus the resolved window and priorWindow bounds. These are the SAME numbers the weekly Slack digest cites, so you can reconstruct a digest's headline metrics (total errors, avg cost, avg duration and their deltas). For a single provider lane's per-run numbers use get_prompt_stats or get_prompt_benchmarks instead. Pass start and end as yyyy-MM-dd.
list_sub_promptsList the competitor sub-prompts under one parent prompt, each with its embedded targetProvider (id, slug, name, url). use get_prompt_benchmarks for the same lanes plus full run history, add_competitor/remove_competitor to change them.
request_runTrigger a new eval run for a prompt. creates a pending eval_request the queue picks up. use list_prompts to find prompt ids. an archived prompt runs too (archiving only takes it off the schedule). an empty body re-runs the prompt with its stored defaults (runner pin, else the parent's, else the org default, else the platform default); pass harnessModel to override, urlMap to reroute hosts, and templateArgs whenever the prompt declares template vars, in its text or its setup command (a run without them is rejected). optional scheduledAt (strict ISO-8601 datetime with Z/offset) delays queue claim until that time; omit for immediate. from a terminal or CI: evals runs request --prompt-id <id> (install: https://2027.dev/evals/cli-install).
create_promptCreate a prompt and request its first eval. With parentPromptId, create an independent scenario: omitted text copies the parent as-is; scenarios cannot have template variables; null setup/login/env inherits, an empty setupCommand ("") disables setup and loginHints: false disables persona instructions (loginHints) (on any prompt), and an empty env map ({}) withholds the parent env (envMode none); a non-empty map replaces it. Scaffold copies the current parent default (else the target provider's). Scenarios inherit the parent schedule. harnessModel pins the runner (harness + model) for this prompt's scheduled runs, overriding the org default set on the settings page — the same pin update_prompt_config sets later; omit or null = no pin (the org default applies), and a scenario copies the parent's pin unless you pass one. checks is this prompt's custom verifier checks, seeded here and editable later with update_prompt_config; on a scenario it cascades at request time like env (null inherits the parent's map live, {} withholds it, a non-empty map replaces it), so omitting it on a scenario inherits rather than copying: a flat map of {"check-slug": "what the judge should look for"} (slugs /^[a-z0-9][a-z0-9_-]*$/) that merges ON TOP OF the verifier's default checks — a non-empty string adds that check or overrides the default of the same slug, null as the VALUE removes that default check, and an empty string is rejected; omit it for no custom checks. The value is judge-prompt input, the text the scoring model reads to decide pass/fail, not display copy: write each as ONE observable outcome that either happened or did not, with no "and/or", "or equivalent", partial-credit or if-blocked clauses and no check that passes because the agent stopped at a blocker — checks show what really happened, not something to pass. The same goes for endGoal and text: simple, concrete, no escape clauses, no hints at the answer; the dashboard shows only the slug and its 0|1 result. checks are not secret and read back verbatim on get_prompt and list_prompts. the queue reads the map at run time, so a later edit changes how FUTURE runs are graded and past runs keep only their {slug: 0|1} results. env values are set as environment variables on the CODING AGENT's own process (and the shells it spawns); no .env file is written in the sandbox, and setupCommand runs earlier — as the sandbox healthcheck, before the agent's environment exists — so a setup script cannot read them. Returns promptId, evalRequestId, evalRunId, and loginHintsStatus (never secret values). Root prompts with unfilled template variables do not run until values are supplied. Archived prompts are never scheduled but still run when requested: status: "archived" writes the prompt row WITHOUT firing a first eval (evalRequestId and evalRunId are null); request_run runs it later. passing parentPromptId of an archived prompt is refused with "Parent prompt <id> is archived" before any write.
update_promptUpdate a prompt's scalar fields (text, title, end goal, status). editing text is blocked once published reports are linked, unless the prompt is still 'suggested' (an unsaved draft) or allowPublishedTextEdit is true (title/end goal/status stay editable). status moves a prompt between active, suggested, and archived — promoting a suggested prompt to active is how it starts running on its cadence. at least one field must be provided.
update_prompt_configConfigure a prompt's eval cadence (runs per eval, frequency), sandbox config, scaffold, persona instructions (loginHints), custom verifier checks, and additionalUrls. cadence, env, setupCommand, scaffold, and loginHints may be set per competitor sub-prompt; additionalUrls is stored only on a top-level prompt and does not create competitor lanes (use add_competitor). env values are write-only (omit env = keep; null = clear this prompt's stored env so sub-prompts re-inherit parent at next snapshot — not an opt-out of parent secrets; object = upsert strings / null tombstone per key; response names only). at run time the resolved env is set as environment variables on the CODING AGENT's own process (and the shells it spawns) — no .env file is written in the sandbox, and setupCommand runs earlier, as the sandbox healthcheck before the agent's environment exists, so a setup script cannot read the prompt's env values. scenarios (child prompts) with null env / setupCommand / loginHints inherit the parent's values at eval-request snapshot time (resolved_env / resolved_setup_command / resolved_login_hints); an explicit empty setupCommand ("") or loginHints: false disables that field instead of inheriting; an empty env map ({}) is stored as-is on a scenario and withholds the parent env (envMode none; a patch-mode {} has no keys to patch and changes nothing; roots collapse {} to null); a non-empty map replaces the parent's entirely (no key merge). scaffold picks the starter codebase the sandbox is built from (react, nextjs, blank, localstack, powersync, user-docs, polyglot): omit=keep, value=set, null=clear on a root (its provider scaffold then applies at run time) or copy the parent's effective scaffold on a scenario (a snapshot against the scenario's own target provider, not live inheritance); changing it rebuilds the sandbox image on the next run. loginHints is the persona instructions the browser agent gets for this prompt: omit=keep, string=set, null=clear (a scenario then inherits the parent's, a top-level prompt its provider's), false=off: no persona instructions for this prompt, scenario or top-level ("" is accepted too); the response returns loginHintsStatus (set | inherited | none) — the resolved status, never the text; reads (get_prompt, list_prompts, get_trace) return a copy with detected secrets masked, never the raw text. additionalUrls is stored only; competitor lanes come from add_competitor. checks is this prompt's custom verifier check map, {"check-slug": "what the judge should look for"}, merged on top of the verifier's defaults: omit=keep the stored map, null=clear it, {} = store an explicitly empty map, and a NON-EMPTY map replaces the stored one wholesale (not a per-key patch — send every check you want kept). on a scenario the stored value cascades at request time like env: null inherits the parent's map live, {} withholds it, a non-empty map replaces it, so null on a scenario means inherit again, not none. a non-empty string adds or overrides the default of the same slug, null as the VALUE removes that default check, "" is rejected, slugs match /^[a-z0-9][a-z0-9_-]*$/. the queue reads the map at run time, so an edit changes how FUTURE runs are graded; past runs keep only their {slug: 0|1} results and cannot be re-read against the new text. checks are not secret and read back verbatim. write each check as ONE observable outcome that either happened or did not: no "and/or", "or equivalent", partial-credit or if-blocked clauses, and no check that passes because the agent stopped at a blocker or reported it — checks show what really happened, not something to pass.
add_competitorAdd a single competitor to a top-level prompt, by url (resolves or creates the provider) or by targetProviderId (an existing provider — ids from list_competitors or get_prompt_benchmarks; exactly one of the two): always inserts a new active competitor sub-prompt and fires its benchmark eval. existing lanes for the same provider are left untouched. the prompt must reference your service and have no unfilled template vars, and must not be archived — an archived parent is refused with "Parent prompt <id> is archived" before any write.
remove_competitorRemove one sub-prompt from a prompt by archiving it (reversible; run history is kept): a competitor lane or a same-provider scenario. pass the sub-prompt id from get_prompt_benchmarks lanes or list_sub_prompts; refuses a top-level prompt.
link_repoLink a GitHub repo ("owner/name") to a prompt so pull requests on that repo get this prompt evaluated against their preview deployment. linking also routes every domain the prompt names — the hosts of urls in its text, plus your org domain — to the repo, adding only domains that have no route yet (existing routes are kept); the response returns the resulting domainRouting; get_prompt lists the linked repos (linkedRepos). linking an already-linked repo changes nothing and succeeds. the repo must be one the firegoat GitHub App is installed on for pull requests to be evaluated.
unlink_repoUnlink a GitHub repo from a prompt: its pull requests stop getting this prompt evaluated. the prompt's domainRouting is left as it is. a repo that is not linked to the prompt is reported as not found.
list_runsList every run your org triggered (newest first), including pending/running/failed ones, with the submitted request config (harness/model/urlMap/templateArgs) and loginHints (whether that run was handed persona instructions; the text is never returned). resolvedPrompt is the screened text that run actually ran with (secrets masked); prompt.text is the live template. report.reward is the verifier's per-check checklist on every outcome (null when none was recorded); its evidence cites raw session uuids, which get_run_trace returns resolved to turn indexes as checkEvidence. outcome (succeeded / goal_not_met / …) and reportSlug sit at the top level — status is 'completed' for both succeeded and goal_not_met. a failed run the platform retried points at the new run in retriedAs (attempt counts from 1, retryOf points back). use this for run status and lifecycle. for scored RESULTS with grades, use list_reports instead. from a terminal or CI: evals runs list (install: https://2027.dev/evals/cli-install).
get_runGet one run's lifecycle status, its outcome (succeeded / goal_not_met / … — status is 'completed' for both) and reportSlug, plus a summary report (null until a report is published). a failed run the platform retried names the new run in retriedAs — follow it for the request's result. report.reward is the verifier's per-check pass/fail checklist (null when none was recorded) — graded on every outcome, so on a goal_not_met run it shows which checks failed. resolvedPrompt is the screened text this run actually ran with (secrets masked); prompt.text is the live template. setupCommand is the setup script this run actually ran (secrets masked; null when it had none) and healthcheckLog is that script's own output (null when the run had no setup command or uploaded nothing; an empty string means the script ran and printed nothing, e.g. a silent command -v x >/dev/null under set -e — read setupCommand for the check that failed) — when a run failed during setup it produces no report and no agent session: this tool and get_run_trace (by runId) both return its setup command and output. for the full metrics/friction/timeline detail, call get_report (GET /reports/{slug}) with that slug. while a run is still going the response may carry lastRunDurationMs — how long this prompt's last completed run took; add it to startedAt for an expected finish, and wait until then before calling again instead of polling. from a terminal or CI: evals runs get --run-id <id> (install: https://2027.dev/evals/cli-install).
list_reportsList published RESULTS — including competitor benchmark reports on your prompts — with their outcome and run identity. for run lifecycle (pending/failed) and submitted config, use list_runs instead. the provider filter only matches competitor reports, which exist only once a competitor sub-prompt has run: an empty result is not an access denial — call list_competitors to see which competitors exist, and add_competitor({ promptId, url }) to create a missing one.
get_reportGet full details for a published/competitor report by slug (hard metrics, timeline, recommendations, summary, the legacy key finding and friction points under legacy, and reward: the verifier's per-check pass/fail checklist on every outcome, goal_not_met included; null when none was recorded). reward.evidence cites raw session uuids; get_trace returns them resolved to turn indexes as checkEvidence. direct fetch works for any report status. metrics.errors and metrics.interruptions are judge-adjusted (provider-caused errors only; half-weight browser login approvals), so they differ from get_trace's raw totalErrors / totalInterruptions; metrics.cost is the run's cost of record.
get_traceGet the agent session conversation log for a published report, by its report slug. for a run that has no report (failed / orphaned, or not yet published), use get_run_trace with the run id (from get_run / list_runs) instead. paginate with offset+limit (default: last 20 turns), or pass all to return every turn. totalToolCalls and toolCounts always cover the whole session, so limit: 1 is the cheap way to get them. each tool result's text is capped at 10,000 chars ("…[truncated N chars]" marks the cut); pass fullToolResults: true for the full text. also returns the run's setup command and healthcheckLog, its output. a failed run whose agent never started (e.g. setup failed) returns sessionRecorded: false with no turns and its failureReason — the setup command and healthcheckLog say why. includeBrowser interleaves browser-agent screenshots whenever the run uploaded any. totalErrors / totalInterruptions are raw counts over the whole session, not get_report's judge-adjusted metrics.errors / metrics.interruptions. checkEvidence maps each verifier check to the turn indexes it cited (reward.evidence's uuids, resolved), so read those turns instead of the whole trace.
get_run_traceGet the agent session conversation log for a run by its run id (from get_run / list_runs), visible to the run's own org (or any org for an admin caller). works the same whether or not the run has a published report. paginate with offset+limit (default: last 20 turns), or pass all to return every turn. totalToolCalls and toolCounts always cover the whole session, so limit: 1 is the cheap way to get them. each tool result's text is capped at 10,000 chars ("…[truncated N chars]" marks the cut); pass fullToolResults: true for the full text. also returns the run's setup command and healthcheckLog, its output. a failed run whose agent never started (e.g. setup failed) returns sessionRecorded: false with no turns and its failureReason — the setup command and healthcheckLog say why. the response's slug is the real report slug when one exists, and null for a run with no report. includeBrowser interleaves browser-agent screenshots whenever the run uploaded any. checkEvidence maps each verifier check to the turn indexes it cited (reward.evidence's uuids, resolved), so read those turns instead of the whole trace.
get_run_artifactsGet download links for a run's stored files by its run id (from get_run / list_runs): session is the agent session.jsonl, with every detected secret masked the same way get_run_trace masks it, and code is code.tar.gz of the agent's working directory. each is a presigned GET url valid for 15 minutes (expiresAt) with its size in bytes, or null when the run has no such file stored (a failed run may have neither). visible to the run's own org (or any org for an admin caller). from a terminal or CI: evals runs get-artifacts --run-id <id>.
cancel_runCancel a run that has not started yet: a pending run (queued, or scheduled for later via scheduledAt) is marked failed with failureReason 'cancelled' and the queue never picks it up. a running or finished run is refused. run ids come from request_run / list_runs.
list_fetched_urlsList the urls agents fetched, the web searches, mcp calls, cli commands and file reads/writes across your organization's eval runs (the Fetched URLs page): per row its hits, outcome breakdown and the runs behind it, most hits first. optional promptId scopes to runs linked to one prompt ("unlinked" = runs linked to none). limit caps each section (null = every row); the *Count fields say how many matched.
list_commentsList the comment thread on a recommendation or a run, oldest first. entityId is a recommendation id from the insights board, or for runs the report slug (from list_reports / get_report / get_run's report.slug). a target with no comments returns an empty list.
add_commentAdd a comment to a recommendation or a run. entityId is a recommendation id from the insights board, or for runs the report slug (from list_reports / get_report / get_run's report.slug). returns the created comment.
list_insightsList your organization's insight cards from the insights board — AI recommendations from eval runs plus manually created ones — with status/effort/assignee and each card's comment thread. optional runSlug filter scopes to one run's insights. paginate with limit/offset. card ids are the entityId for add_comment/list_comments (entityType "recommendation").
create_insightCreate a new insight card on the org's insights board (not sourced from an eval run). optionally set effort, estimated impact, and its external-origin source. returns the created card with its (empty) comment thread.
update_insightUpdate an insight card: edit title/content/effort/estimatedImpact, set or clear its external-origin source (null clears it), set status (open/accepted/dismissed/fixed), or assign it (kind "user" needs ref = a member's user id; null unassigns). pass orderedIds to reorder the destination column when moving between statuses. at least one field must be provided.
send_feedbackSend feedback about the platform to the 2027 team: a bug report, a feature request, or anything confusing. call it (type bug) the moment a platform call fights you: a misleading or missing error, a response that contradicts another tool, a write that returns success but doesn't stick on a later read, input silently dropped, a documented field that isn't there. include the request you sent, the response you got and what you expected; title is a short one-line summary shown in the team's triage list; put report slugs / run ids / tool names in context so the team can reproduce. then continue — do not retry the same call more than once. confusing, slow, or guesswork is type feedback; something you wished existed is type feature_request. skip only your own mistakes and one-off transient errors. to discuss a run or recommendation with your own team, use add_comment instead.
list_feedbackList the issues agents filed against your organization while running evals (bugs, feature requests, feedback), newest first, with each issue's title, body, status (open / acknowledged / resolved) and what the agent reported about itself. read one with get_feedback.
get_feedbackRead one issue an agent filed against your organization, by id, including who acknowledged or resolved it and when.
update_feedbackSet the status of an issue an agent filed against your organization: acknowledged (you've seen it), resolved (it's fixed) or open (reopen it). records who changed it and when, as get_feedback shows.
list_competitorsList the competitors your org is benchmarking, derived from the active competitor lanes across your prompts, with the lanes targeting each. optional search filters case-insensitively over slug, name, and url. an empty list means this org has no competitor lanes yet, not that access was denied — create the first with add_competitor({ promptId, url }), passing a top-level prompt id from list_prompts and the competitor's homepage url. remove one with remove_competitor.
get_providerLook up the provider a domain belongs to and get its id, name and domains. pass any domain the provider owns (an alias like clerk.dev, or www.clerk.com, resolves too). use the id as targetProviderId. not found means no provider owns that domain yet; add_competitor({ promptId, url }) creates one.
list_credentialsList your org's named credentials. sensitive credentials return their character count only (the value is never returned); non-sensitive ones include the value.
add_credentialAdd a named credential to your org. names are unique per org (case-insensitive). sensitive (default true) hides the value from every read; the response carries its length instead. targetProviderId scopes it to one provider (your product or a competitor); null = usable from every prompt in the org; never injected on its own: copy it into a prompt's env with envCredentials on create_prompt / update_prompt_config.
edit_credentialEdit a credential's name, value, sensitive flag, or targetProviderId (null = usable from every prompt in the org; never injected on its own: copy it into a prompt's env with envCredentials on create_prompt / update_prompt_config; omit to keep). passing value replaces the stored secret; the old one is never returned.
remove_credentialPermanently delete a credential from your org.
list_briefsList your organization's briefs — period reports over the org's published runs — newest edit first, with each brief's period, status, frozen run count and averaged metrics. published briefs belong to the whole organization; drafts are listed only to whoever created them. bodies are not included; read one with get_brief.
get_briefRead one brief by slug, including its markdown body. pass version to read a published snapshot instead of the live body; omit it for the live one. a draft is readable only by whoever created it — someone else's reads as not found.
list_brief_versionsList a brief's published versions, oldest first, with who published each and when. read a version's body with get_brief(version).
create_briefCreate a draft brief over a date period (day/week/month/custom). the org's published runs inside the period are frozen into the brief's scope and their metrics averaged; the slug is derived from the period. the body starts as section scaffolding — write it with update_brief.
update_briefEdit a brief: replace its markdown body, set or clear (null) its title, or archive it / return it to draft with status. cite a run with [[mention:<report-slug>]] inline in prose — an inline link to the report, and the only run citation to use. unresolved refs are allowed while drafting and refused by publish_brief.
publish_briefPublish the brief's current body as its next version and make it readable at /{org}/briefs/{slug}. refused while the body cites runs or items that do not resolve.
whoamiShow who you are calling as: your email, the org you act for, that org's provider, and whether you authenticated with an API key or a signed-in session. call it to check which account and org a key or login belongs to.
create_sandboxCreate a Linux sandbox (bash, internet access) and return its sandboxId. it belongs to you alone: create one only when you need to run a command, then reuse the sandboxId you created for every run_command; files persist in it. call remove_sandbox when you are done. a sandbox is removed after 15 minutes without a command and lives at most 1 hour.
run_commandRun a bash command in a sandbox (reuse the sandboxId you created with create_sandbox). returns exitCode, stdout and stderr, each cut to its last 10000 bytes and decoded as UTF-8, plus stdoutLength / stderrLength: the total bytes each stream produced. a command is killed after 120 seconds and returns timedOut: true with exitCode null and the output so far; start a long job in the background with nohup cmd >/dev/null 2>&1 &. each command extends the sandbox's 15-minute idle timeout; a sandbox lives at most 1 hour, then create a new one.
remove_sandboxRemove a sandbox you created with create_sandbox when you no longer need it. its files are gone afterwards. a sandbox you forget is removed after 15 minutes without a command, and every sandbox lives at most 1 hour.

request_run creates a pending request that the queue picks up, exactly like the REST …/run endpoint and the dashboard's manual trigger; the other write tools (create_prompt, update_prompt, update_prompt_config and add_competitor) manage prompts the same way the dashboard does, and send_feedback files a bug report or feature request straight to the Firegoat team. Note the run/report split: list_runs/get_run cover the run lifecycle (pending → completed) and the config you submitted, while list_reports/get_report cover the scored results. The read tools make it easy to ask an agent things like "how did our last run score?" or "what friction did the latest report hit?".

When something looks wrong

If a call fights your agent — a misleading or missing error, a response that contradicts another endpoint, a write that returns success but doesn't stick on a later read, input silently dropped, a field this page promises that isn't there — have it call send_feedback (or POST /api/v1/feedback) with type bug while it's fresh, then continue. Include the request sent, the response received and what was expected, with API keys, Authorization headers and any other secrets removed; put report slugs, run/prompt ids and tool names in context. Don't retry the same call more than once. Broken or contradictory is bug; confusing, slow, or guesswork is feedback; wished it existed is feature_request. Skip only the agent's own mistakes and one-off transient errors. A minimal POST /api/v1/feedback body:

{
  "title": "GET /runs/{runId} returns status completed with report null",
  "type": "bug",
  "body": "Expected the report once status is completed. Got status: completed, report: null on three consecutive reads a minute apart.",
  "context": "runId 5f1c…, prompt p_9a2…, called via get_run"
}