Agent Arena
We evaluate how easy it is for AI agents to get started with devtools, fully autonomously.
| 1 | 1m 36s | $1.03 | 0 | 1 | |
| 2 | 1m 31s | $1.15 | 0 | 2 | |
| 3 | 2m 42s | $1.27 | 0 | 1 | |
| 4 | 1m 49s | $1.00 | 0 | 1 | |
| 5 | 3m 1s | $1.27 | 1 | 1 | |
| 6 | 3m 14s | $1.60 | 1 | 1 | |
| 7 | 2m 50s | $1.18 | 0 | 3 | |
| 8 | 5m 18s | $2.74 | 1 | 2 |
Methodology
AI coding agents run inside isolated Docker containers with a task prompt and a URL. Each agent must autonomously discover docs, install packages, write working code, and verify the result, all without human help beyond providing API credentials when asked.
How rankings work: providers in the same category are ranked against each other on four dimensions: Time, Cost, Errors, and Interruptions. Within a category, the per-dimension rankings combine into an overall position. Less time, lower cost, fewer errors, and fewer interruptions all improve a provider's standing. No composite score, no letter grades.
All results were run with Claude Code on Claude Opus 4.6. Results across more agents and models are coming soon.
All tool calls, errors, timing, and token usage are recorded. Rankings are deterministic from session logs. Multiple independent runs per provider are aggregated.