GitHub pull requests

There are two ways to get an AX grade on every pull request: the Firegoat Evals GitHub App, which watches your repos for you, and the self-hostable GitHub Action, which you add to your own CI and drive with an API key.

GitHub App

The App is the zero-config option. It is managed from the Settings and PRs pages.

Connecting. On the Settings or PRs page, click Connect GitHub. That sends you to GitHub's install screen (https://github.com/apps/firegoat-evals/installations/new) carrying your org as state; after you choose repositories, GitHub returns you to the dashboard and the connection is recorded. Repository selection happens on GitHub: the Manage repos on GitHub → link opens the App's installation settings to add or remove repos.

Linking repos to prompts. A repo only produces evals once an active prompt is linked to it. On the PRs page use Link a Repo to a Prompt (or + Create a new prompt) to connect a repo to the prompt that should run against its preview deployments. Linking also auto-derives which domains in the prompt map to that repo. You can do the same over the API (POST /api/v1/prompts/{promptId}/repos) or with the link_repo MCP tool.

What happens on a PR. When a pull request's preview deployment goes live (detected from Mintlify, Vercel or Netlify), the App creates an eval request for each linked prompt, posts a pending commit status, and maintains a single sticky PR comment that updates from Waiting → Running → the graded result (score bar, metric deltas versus the prompt's baseline, top friction points, and report/trace links). Force-pushing supersedes in-flight runs, and if no preview shows up within 10 minutes a helper comment suggests installing a deploy platform. Production deployments are ignored.

GitHub Action

The Setup page (/<org>/setup) walks you through the self-hostable GitHub Action so that every pull request gets an AX grade posted as a sticky comment. The Action authenticates to Firegoat with an API key, polls the run, and posts the comment and commit status from your runner's own GITHUB_TOKEN. The source lives at github.com/firegoat-ai/evals-action.

The guide has four steps:

1. Create an API key and save it as the repository secret EVALS_API_KEY (under Settings → Secrets and variables → Actions on GitHub). The key is shown only once.
2. Select a prompt to evaluate. The page fills its id into prompt-id for you and previews the prompt text and any template variables.
3. Add the workflow at .github/workflows/firegoat.yaml. It triggers on deployment_status (only when state == 'success', i.e. after your preview deploy is live, and not for the Production environment), runs firegoat-ai/evals-action, and needs the runner permissions contents: read, pull-requests: write, statuses: write. The url-map input rewrites the production hostname your prompt names (the key: replace <production-hostname> with it, for example docs.acme.com) to the preview URL GitHub reports (environment_url, else target_url), so the agent tests your preview instead of production.
4. Result: once the eval finishes, a sticky comment appears on the PR saying whether the run succeeded or did not finish, with a metrics table (time, cost, errors, interruptions) and deltas against the prompt's baseline, the template variables and URL map it ran with, and a link to the full report. If the eval is still running when the Action stops polling (wait-timeout-minutes, default 20), the comment stays at Running and the commit status is set to success so the check isn't stuck pending; set timeout-fails: 'true' to make it a failure instead.

A minimal workflow looks like:

name: Firegoat eval
on:
  deployment_status:
jobs:
  eval:
    if: github.event.deployment_status.state == 'success' && github.event.deployment_status.environment != 'Production'
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
      statuses: write
    steps:
      - uses: actions/checkout@v4
      - uses: firegoat-ai/evals-action@v0.8.0
        with:
          api-key: ${{ secrets.EVALS_API_KEY }}
          prompt-id: <your-prompt-id>
          url-map: |
            { "<production-hostname>": "${{ github.event.deployment_status.environment_url || github.event.deployment_status.target_url }}" }

This recipe fits platforms that report through GitHub's Deployments API, such as Vercel and Mintlify. Netlify posts a commit status instead, so it needs an on: status trigger; the Action's README has that recipe and one for running after your own deploy step.

If your prompt uses {{variable}} placeholders, pass them with a template-vars: JSON input (max 50 variables, values up to 2,000 characters).