README testing is cheap agent evaluation for humans
A small Hacker News item about paying people to follow a README points at a bigger builder habit: treat docs, setup flows, and AI-agent instructions as things to test, not things to admire after shipping.
TL;DR: If a stranger cannot follow your README without help, your users, teammates, and coding agents probably cannot either.
What breaks when strangers run your README?
Hacker News surfaced a small, useful idea in the item titled “I paid people to try and follow my README.” That sentence is the whole lesson.
Most teams treat a README like a receipt. Something you attach after the work is done. Install steps, env vars, a sample command, a few screenshots if everyone is feeling disciplined. Then the repo ships.
But a README is not a receipt. It is an interface.
It has state. It has dependencies. It has missing assumptions. It has “obvious” steps that are only obvious to the person who wrote the code last Tuesday. The fastest way to find those gaps is to put the instructions in front of someone who has no context and watch where they stop.
That is especially true now because fewer first runs are done by humans alone. A developer may paste your README into Claude Code, Cursor, Codex, or another coding agent and ask it to get the project running. If your setup instructions are vague, the agent will fill gaps with guesses. Sometimes good guesses. Sometimes package-manager fan fiction.
The same bug shows up in product onboarding, internal runbooks, AI workflow prompts, API docs, and SOPs. The author knows too much. The reader is punished for it.
Why not just ask an LLM to test the docs?
You should. But not only that.
An LLM can act as a first-pass lint tool for documentation. Ask it to read the README as a new user and flag hidden assumptions, missing prerequisites, undefined terms, incompatible commands, and places where the order seems wrong. Then ask it to generate a “first-run checklist” from the docs and compare that checklist to what actually works on a clean machine.
That catches a lot.
It will not catch everything. Models are trained to be helpful, which means they often smooth over ambiguity instead of tripping on it. A human tester with a blank machine, limited patience, and no emotional investment in your repo gives a different signal. They show you the actual friction: “I do not know which file you mean,” “this command failed,” “I thought I was supposed to create the API key earlier,” “I quit here.”
That signal is closer to production than a polished doc review.
For AI builders, this maps cleanly to agent evaluation. We spend a lot of time asking whether an agent can complete tasks. Less time asking whether the task environment is legible. Sometimes the model fails because it is weak. Sometimes it fails because the instructions are underspecified, the repo layout is weird, the error messages are stale, or the success condition is invisible.
If an average human cannot complete the flow from the README, an agent benchmark built on the same flow is not testing autonomy. It is testing tolerance for bad instructions.
What should builders copy from this?
The move is simple: pay or recruit people who are not already inside your project, ask them to follow the README from scratch, and do not help them unless the test design says you can. Watch the screen. Record where they hesitate. Ask them to narrate what they expected to happen.
Then fix the README like you would fix code. Remove dead paths. Add exact prerequisites. Put commands in the order they are used. Say what “done” looks like. Include the most likely failure modes. Test on a clean environment, not the maintainer laptop with three years of global packages and shell aliases.
For AI-assisted development, add one more pass: run the README through the tools your users are likely to use. Give it to a coding agent in a fresh repo context. See whether it can install, configure, run tests, and explain failures without private knowledge. If it invents missing steps, that is not only a model issue. It is a docs issue.
Practitioner’s take: before the next release, pick one critical setup path and run a two-part test. First, ask an LLM to critique the README for missing assumptions. Second, pay one unfamiliar human to follow it on a clean machine while you watch silently. The catch most teams miss is that the README is now part of the runtime for humans and agents. If it is broken, your product is harder to use than you think.