How API Documentation Format Affects AI Code Generation
Every time an AI coding tool reads your API docs, the tool spends tokens. Does the format of those docs affect the number of tokens? The code quality? We ran over 21,000 integration tests across 4 AI models and 2 APIs to find out.
Developers still read your API docs. But increasingly, AI coding tools read them first. Every token your documentation consumes is a token that can't be used for reasoning, code generation, or context about the user's project.
Research updates, chapter releases, and documentation strategy insights. 1-2 emails per month. No spam.
Popular APIs are all over the internet, including in LLM training data. Testing with Stripe's docs means you might be measuring memorization, not comprehension. We built two control APIs from scratch to eliminate contamination.
Identical information documented in four different formats. Same endpoints, same parameters, same constraints. The only variable is how it's structured.
From local models you can run on a laptop to frontier cloud APIs. Format impact changes with model capability.
Token efficiency, code generation pass rates, error handling quality, and overall accuracy. Not just "does it work" but "how well, at what cost."
The problem nobody's measuring. Why your doc format creates a hidden tax on every AI interaction with your API.
Over 21,000 integration tests. 4 formats. 4 models. 2 APIs. The complete results, with data on token efficiency, code generation quality, and error handling.
The decision framework. How to choose the right format for your team, test your own docs, and optimize without rewriting everything. Where documentation is heading. Plus the complete testing toolkit, the control API references in all formats, and everything you need to reproduce the research yourself.
Paste your API documentation and see the token cost in real time. The first step to understanding the problem.
Try the Docs Cost Calculator →A detailed look at the book's structure, the research included in each part, and the key takeaways for API teams.
After several months and 21,000+ integration tests, Tokens Not Jokin' is here.
If you test how well AI tools use Stripe's API docs, how do you know what you're measuring? Here's how I solved the contamination problem.
Your company tracks API latency, uptime, and error rates. But nobody's tracking what it costs AI tools to read your API documentation.
Available now on Leanpub
Available on Leanpub. All formats: PDF, EPUB, and web.
New findings, chapter releases, and documentation strategy insights. No spam, no fluff.