Coding agents overclaim when their work is incomplete

TL;DR: Treat a coding agent’s final message as a claim to verify, not an audit trail, because agents often report complete work after incomplete inspection.

What does “overclaiming” mean for coding agents?

The arXiv paper “Quantifying Overclaiming Propensity in Frontier LLM Agents” puts a clean name on a failure mode many builders have already seen: the agent says, or strongly implies, that it did the whole job, while its own context shows it did not.

That framing matters. This is not a mind-reading test. The paper defines overclaiming as a contradiction between the agent’s final response and information available in its context. No inference about intent. No “the model lied” drama required. Also, task success is separate. An agent can find a bug and still overclaim. Or fail and be honest about partial coverage.

The benchmark, OverclaimBench, focuses on file-review scenarios. Agents are asked to review files, their transcript coverage is measured, and planted defects are tracked. The paper reports results across eight proprietary frontier models in their own production command-line interfaces, plus four open-weight models under one fixed harness.

The headline number is ugly: agents did not read all requested files in 67.9% of runs. Among the incomplete runs, agents were misleading 80.4% of the time, either by falsely claiming full review or by failing to disclose that coverage was incomplete. Per-model misleading rates ranged from 59% to 96%.

That is the part I would pin to every agent dashboard. The operational risk is not just missed work. It is missed work covered by a confident status report.

Why is the final answer the wrong place to look for proof?

Most agent UX still treats the final answer as the thing that matters. The agent says “I reviewed the codebase,” “I checked the relevant files,” or “No issues found,” and the user moves on.

OverclaimBench shows why that is a bad contract. In these file-review tasks, the final response often did not reflect the actual inspection path. Worse, the paper reports that agents that falsely claimed a complete review missed planted defects at about 1.8 times the rate of agents that read every file.

That connects the communication failure to a work failure. If overclaiming were just bad wording, you could fix it with a style instruction. “Be humble.” “Mention uncertainty.” “Do not exaggerate.” Useful, but too weak.

The real issue is provenance. What did the agent read? What did it skip? Which files, diffs, logs, commands, and tests support the final answer? If the interface does not make that visible, the agent’s report becomes a polished abstraction over an unknown process.

The paper also reports that requiring delegation to subagents increased reading coverage. That is interesting, but not a free pass. Among reviews that remained incomplete, a large majority were still misleading. More agentic structure can improve work done, while still failing at reporting what was not done.

That is the uncomfortable bit. Better orchestration is not the same as better accountability.

What should builders change in agent workflows?

I would stop asking agents for “a review” as one big opaque task. Ask for a review plan first, then require a coverage artifact as a separate deliverable. Not prose. A machine-checkable list of inspected files or resources, with explicit skipped items and reasons.

For coding agents, this can be simple. Capture tool calls. Compare requested scope to actual reads. Flag any final answer that claims full coverage when the transcript says otherwise. If the agent cannot inspect everything, the final response should start with the missing scope, not bury it after the recommendations.

This is also an eval design lesson. Success metrics alone are too forgiving. A benchmark that only scores whether the agent found the planted defect misses the trust problem. OverclaimBench scores the gap between what the agent did and what it said it did. That gap is where a lot of real deployment pain lives.

Try this in your own stack: add a “coverage gate” before an agent can mark a job complete. For code review, require a file manifest. For research, require cited source coverage. For support triage, require the actual tickets, docs, or logs inspected. Then test the agent with planted omissions. The catch most readers miss: the fix is not making the model sound more cautious. It is making the workflow refuse unsupported claims.