REST API

Authentication

The public REST API is org-scoped and lives under https://firegoat.ai/evals/api/v1. Authenticate every request with Authorization: Bearer $EVALS_API_KEY. Responses are JSON: list endpoints return a bare array, single-resource endpoints return a bare object, and errors look like { "error": "...", "message": "...", "code": "...", "issues": [...] } (error is the human-readable message; code is the error kind, e.g. UNAUTHORIZED/BAD_REQUEST/NOT_FOUND; issues is present only on input-validation failures). Because access is keyed to your org, resources outside it return 404, not 403.

Endpoints

Endpoints that take no body

A few writes take their whole input from the path — archiving a competitor lane or scenario (PATCH /api/v1/competitors/{subPromptId}) removing a credential (PATCH /api/v1/credentials/{credentialId}/remove) and removing a sandbox (PATCH /api/v1/sandboxes/{sandboxId}/remove). Send no body, or {} with Content-Type: application/json — an empty body reads as {}.

Triggering a run

POST …/run takes a JSON body, and {} (or no body at all) is a complete request: the run falls back to the prompt's stored defaults, picking the runner from the prompt's own pin, then its parent's, then your org default, then the platform default. The one exception is a prompt that declares {{variables}} — it still needs templateArgs, and a request without them returns 400. urlMap rewrites hostnames so the agent hits a preview or staging deploy instead of production; each key is the production hostname to intercept and each value is the full replacement URL (must be http(s), max 100 entries). templateArgs supplies values for the prompt's declared {{variables}} (max 50, each up to 2,000 characters). Missing or unknown template variables return 400. harnessModel (an object with harness and model) pins the run to a specific agent harness and model; unknown combos return 400.

curl -X POST \
  -H "Authorization: Bearer $EVALS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "urlMap": { "docs.acme.com": "https://preview-abc123.vercel.app" } }' \
  https://firegoat.ai/evals/api/v1/prompts/<promptId>/run

The response gives you a runId, a statusUrl to poll, and a runUrl linking to the human report page:

{
  "runId": "…",
  "statusUrl": "https://firegoat.ai/evals/api/v1/runs/…",
  "runUrl": "https://firegoat.ai/evals/<org>/runs/…"
}

Poll statusUrl until status is completed or failed; both are final. A completed run carries its outcome (succeeded, goal_not_met, …) and a reportSlug, and its report summary fills in once the report is published. Pass reportSlug to GET /api/v1/reports/{slug} for the full report (metrics, dimensions, timeline). A failed run has no report: read its failureReason, and if the platform retried it, retriedAs names the new run to poll instead. From a terminal, the evals CLI wraps the same calls: evals runs request --prompt-id <prompt-id>.

OpenAPI spec

The full machine-readable spec (OpenAPI 3.1) is served as JSON at firegoat.ai/evals/api/openapi. No auth is required to read it. It describes all v1 endpoints, the bearerAuth security scheme, and the request and response schemas (Prompt, Run, RunRequest and the rest). Point a client generator, Postman, or any OpenAPI tool at that URL to get typed bindings. The Setup page links to it as "OpenAPI spec".

Reporting a problem

If a call misbehaves (a misleading error, a response that contradicts another endpoint, a write that does not stick), send it to us with POST /api/v1/feedback. The MCP server page has the body format and what to include.