On October 21st, Firegoat took the main stage at Resend Forward, Resend’s first-ever conference.
We had the honor of sharing the main stage with Guillermo Rauch (CEO, Vercel), Cristina Cordova (COO, Linear), Calvin French-Owen (EIR, Thrive Capital), Zeno Rocha (CEO, Resend) and Aaron Francis (Laravel). My talk was about the new power user of every developer product, the agent, and what it takes to win it over. Here are the key ideas.
Agents are already your power users
Agents show up in your docs first. Mintlify, which hosts docs for a huge share of developer tools, reported that 66% of measured docs traffic in August came from AI agents, up from about 21% in February.
And they don’t stop at reading. They deploy, provision and sign up. Guillermo Rauch shared that 60% of Vercel deployments are now triggered by agents, up from 3% in January. When Databricks acquired Neon, over 80% of Neon databases were created by agents. Supabase says more than 60% of new databases are started by an AI coding tool.
For years, DevRel and product teams worked to win the hearts and minds of developers. That still matters, but developers increasingly delegate. The agent picks the tool, reads the docs and writes the integration. So now we have to win over their agents too.
A new metric: value per token
For humans, we measured time to first value. For agents, every step costs tokens: every docs page read, every retry, every wrong turn. So the metric I’d put on the wall is value per token: how much an agent actually gets done with your product for every token it spends. A tool that gets the job done in 20 thousand tokens beats one that needs 200 thousand, and agents and their users will notice.
Our key findings
At Firegoat, our whole job is watching coding agents try to use developer tools: sign up, read docs, write code and recover from errors. Since February, we’ve run more than 10,000 evals on over 150 developer tools. Here’s what the last 90 days alone, about 7,000 runs on more than 100 of those tools, tell us.
- Docs come first. 60% of agent friction starts with a docs problem.
- Agents read docs, not search results. Only 8% of runs used web search. Agents go straight to your docs, and 70% of the pages they open are docs pages.
- Agents want markdown. 46% of runs that read the web pulled a raw .md page or an llms.txt instead of the HTML page. Serve markdown versions of your docs.
- Friction costs you the run. Failed runs had twice the interruptions of successful ones.
- Errors can steer agents. An agent takes your error message literally and acts on the exact text. Yet 16% of errors say what broke but not what to do next. Every error is a chance to tell the agent which command to run, with a link.
Your new users can test your product any time
Here’s the good news. With human developers, user research is slow: you recruit people, schedule calls and watch a few sessions. Agents are right here. You can spin up ten of them in a minute, give them a real task, and they’ll show you exactly where your product falls short. There are two ways to do it: run evals, and ask the agents directly.
Run evals. Learn. Improve. Repeat.
The best teams treat evals as a loop, not a one-off audit. Run real agents on your most critical scenarios, learn where they get stuck and why, ship the fixes, and measure whether value per token actually went up. Then do it again on every release. Here’s what we’ve learned about doing it well:
- Use the data you already have. Support tickets, usage data, MCP and CLI telemetry, agent traffic on your docs, and the questions people ask your docs tell you what matters.
- Start with your critical workflows. For almost everyone, that’s onboarding: a new app, adding you to an existing app, or migrating from a competitor.
- Differentiate by scenario. Run with an API key and without one. Without a key, you see the real first run: signup, auth and provisioning.
- Set goals and measure. Track time, tokens and value per token on each workflow.
- Make AX part of building. Run evals on every preview, before anything goes live.
- Alert on regressions. Watch your critical workflows so nothing breaks in production.
The best teams go where agents get stuck
The best teams don’t stop at an MCP server and an llms.txt. They go to every place an agent might look or get stuck. We wrote this up with the Vercel team, and four patterns stood out:
- Docs where the agent already looks. An 8KB docs index in AGENTS.md took Next.js evals from 53% to 100%, while the same docs shipped as a skill went uncalled in 56% of runs.
- Smart 404s. Agents guess URLs, so a missing docs page answers in markdown with the pages it probably meant.
- Non-interactive by default. The CLI detects coding agents, skips prompts, and returns JSON errors with the exact next command to run.
- Deploy first, sign up later. Sign-up is where agents fail most, so
vercel deploywith no account puts a site live and prints a claim link.
Agents are ready to give feedback
Evals tell you where agents fail on the scenarios you designed. The newer insight is that real agents, doing real work for your users, are ready to tell you what broke. Picture a typical session: the CLI reports success, nothing was actually created, and the agent quietly switches to a competitor. Your team never hears about it.
Your MCP server and CLI could capture two things here: telemetry, meaning what agents actually ran and what failed, and feedback, meaning why it failed, in the agent’s own words. Today almost nobody captures either. That’s the opportunity. Before you build it, get these six things right:
- Be where agents look. A feedback command nobody finds is useless. Put it in your docs, CLI, MCP server, AGENTS.md and error messages.
- Stay compliant. Agents are trained to resist anything that looks like prompt injection. Make the ask explicit, scoped and clearly in the user’s interest.
- Test for reproducibility. A report is a hypothesis. Run an eval to see whether the issue is real.
- Prioritize. LLMs still struggle to judge importance, so use your own judge to classify what’s critical. If you already run evals, you know which workflows matter.
- Route it to the right team. Feedback only helps when it lands with the people who own the fix.
- Let an agent pick it up. Hand the issue to another agent to work on, so humans are only there to merge.
Every signal feeds one loop
Put it together and it’s the same loop as before: run evals, learn, improve, measure. What feeds it is everything you already have: usage data, support tickets, MCP and CLI telemetry, docs traffic and questions, and your team’s intuition. Agent feedback is a new addition to that list, not a replacement. Reproduce each report and it becomes another eval scenario.
Try it from your repo
You can run an AX eval right from your repo today. npx firegoat init sets up a firegoat.yaml where you describe your tests (the easiest way is to let your own coding agent write them). Then npx firegoat eval gives a fresh Claude Code each prompt in an empty folder, saves the session and runs your checks, all on your machine with your own Claude Code login. It’s an early preview, so I’d love your feedback. Learn more about the CLI.
Building is easy. Building well is not.
With the velocity we have now, it’s incredibly easy to build software. It’s still hard to build intentionally designed software that works equally well for humans and for agents. Evals are how you keep that intention while moving this fast, and that’s why Firegoat exists: to empower teams to ship agent-ready products.