Benchmarks
Agent Experience (AX) Tier List
This list asks one question: can an AI agent start using a service without a human in the loop?
36 of 36 providers
| Details | Provider | Tier | Discovery | Login flow | Bot protection | Paywall | Sign-up |
|---|---|---|---|---|---|---|---|
SHands-off | No credential needed, nothing to find | Library, no account | None | Freemium, no card | Not needed | ||
SHands-off | Found | Claim CLI, no account | None | Freemium, no card | Not needed | ||
SHands-off | Found | x402 gateway, no account | None | Pay-per-call, no card | Not needed | ||
SHands-off | Found | auth.md provision + claim | None | Freemium, no card | Not needed | ||
SHands-off | Found | Claim, no account | None on claim path | Freemium, no card | Not needed | ||
SHands-off | Only when pointed | Anonymous deploy | None | Freemium, no card | Not needed | ||
SHands-off | Only when pointed | Puzzle key | None | Freemium, no card | Not needed | ||
SHands-off | Only when pointed | Pre-provisioned scaffold + claim | None | Freemium, no card | Not needed | ||
SHands-off | Only when pointed | Claim, no account | None | Freemium, no card | Not needed | ||
SHands-off | Never found | x402 gateway, no account | None | Pay-per-call, no card | Not needed | ||
SHands-off | Never found | x402 gateway, no account | None | Pay-per-call, no card | Not needed | ||
SHands-off | Never found | x402 gateway, no account | None | Pay-per-call, no card | Not needed | ||
SHands-off | Never found | Unauthenticated project + claim | None | Freemium, no card | Not needed | ||
SHands-off | Never found | Claim, no account | None | Freemium, no card | Not needed | ||
SHands-off | Never found | Claim CLI, no account | None | Freemium, no card | Not needed | ||
SHands-off | Never found | x402 gateway, no account | None | Pay-per-call, no card | Not needed | ||
SHands-off | Never found | x402 gateway, no account | None | Pay-per-call, no card | Not needed | ||
SHands-off | Never found | Anonymous provision + claim | None | Freemium, no card | Not needed | ||
AJust confirm | Found | Headless CLI signup | None | Freemium, no card | Email + password, headless | ||
AJust confirm | Never found | CLI login | None | Freemium, no card | One click | ||
BCopy a key | No better path for an agent to find | One key | None | Card on file | One click | ||
BCopy a key | Found | One key | None | Freemium, no card | One click | ||
BCopy a key | Found | One key | None | Freemium, no card | One click | ||
BCopy a key | Found | One key | None | Freemium, no card | One click | ||
BCopy a key | Found | One key | reCAPTCHA | Card on file | One click | ||
BCopy a key | Found, didn't work | One key | Vercel Security Checkpoint | Freemium, no card | One click | ||
BCopy a key | Found, didn't work | One key | Signup captcha | Freemium, no card | One click | ||
BCopy a key | Never found | One key | None | Freemium, no card | Email + OTP | ||
CFigure it out | Found | ID + secret | None | Freemium, no card | One click | ||
CFigure it out | Found | 4 values | reCAPTCHA | Freemium, no card | |||
CFigure it out | Found | One key | Captcha | Freemium, no card | Approval | ||
CFigure it out | Found, didn't work | PK + SK + host | None on login | Freemium, no card | Google sign-in, one click | ||
CFigure it out | Found, didn't work | Device code, expires 5min | None | Freemium, no card | One click | ||
CFigure it out | Found, didn't work | One key | Sign in + sign up | Not graded | One click | ||
DBlocked | No better path for an agent to find | CLI sign-up | Signup captcha | Enterprise-only API keys | Email + OTP | ||
DBlocked | Found | Connection string | None | Card required | One click |
Methodology
- A single blocking step is enough to stop agentic user activation, so a provider's tier is set by its weakest criterion.
- The Tier column shows the best path a provider offers; the Discovery column says whether the agents in our runs actually found it.
- When a provider offers more than one way in, we grade the easiest one that works for an agent. A captcha or a login wall on a route the agent never has to take does not count against it.
- Runs come from our eval program between June and September 2026, which spans Claude Opus, Sonnet and Haiku, plus GPT-5.6 and Grok harnesses.
- Login flow and paywall are mostly verified from our own runs, while bot protection and sign-up are mostly read from provider documentation.
- Every piece of evidence in a provider's detail row names its source: a Firegoat eval run, the provider docs, or our own check.
- A run only overrides documentation where the agent actually attempted that path.
What tiers mean
| Tier | What it costs you | Login flow | Bot protection | Paywall | Sign-up |
|---|---|---|---|---|---|
SHands-off | User does nothing. Agent starts using the service autonomously | Anonymous or claimable resource, agent-auth (auth.md, x402) | None | No card. Free tier, or the agent pays itself (x402) | Not needed |
AJust confirm | User only confirms actions. Straightforward flow | CLI login (device code, OAuth confirm) | None, or agent-solvable | No card | Google or GitHub, one click |
BCopy a key | User, or their browser agent, does a simple action on the website | One key, copied from a dashboard | Sign up only, one captcha | Card on file | Email + OTP |
CFigure it out | User, or their browser agent, has to think, read, understand how the integration works, or take multiple complicated actions | 2+ keys, multi-step, keys hard to find | Sign in + sign up | Card required to start | Email + verify domain or approval |
DBlocked | User can't start | No self-serve path | Blocks everything | Paid only, sales call | Invite, waitlist, KYC |
What the criteria mean
- Login flow: Can the agent get a working credential itself, or must a human paste in a key?
- Bot protection: Does a captcha or similar challenge stop it?
- Paywall: Can the agent pay its own way, or must a human enter a card or take a sales call?
- Sign-up: Can an account be created without a human?
Don't see your tool?Request an evaluation