Google's Paper Assistant Wants to Catch Your Math Errors Before a Reviewer Does

Peer review is buckling. The volume of submitted papers keeps climbing, a growing share of that work is itself AI-assisted, and the pool of qualified reviewers is not growing to match. Something has to give. Google’s answer, laid out in a new paper introducing the Paper Assistant Tool (PAT), is to point AI at the review bottleneck the same way researchers already point it at discovery.

I want to be careful here because “AI reviews science” is the kind of headline that triggers a reflexive groan. Reviewers are already getting lazy LLM-generated reviews, authors hate them, and conferences are scrambling to ban or detect them. So let me say what PAT actually is and is not, based on what the paper describes.

What PAT actually does

PAT is an agentic framework that ingests a full manuscript, not an abstract or a chunk, and produces a structured evaluation. It checks theoretical results, validates experiments, suggests improvements, and flags potential flaws. The key technical move is inference scaling: instead of asking a model once “is this paper sound,” PAT runs deeper, multi-step verification that catches issues a single call misses.

The number they report is a 34% improvement over zero-shot recall on mathematical errors in the SPOT benchmark. Recall matters here. The job of a pre-submission tool is to catch the error that would otherwise slip through, so missing fewer real errors is the metric that counts. A 34% relative lift on catching math mistakes is meaningful if it holds up outside the benchmark, and that “if” is doing real work.

What I find more interesting than the headline number is the framing. Google proposes a taxonomy of four progressive levels of AI-human collaboration in scientific evaluation. That is a tell. They are not claiming “AI reviewer replaces human reviewer.” They are sketching a ladder, where the bottom rungs are assistive and the top rungs are more autonomous, and they are explicit that PAT lives near the bottom: a pre-submission tool that preserves referee control over outcomes.

Pre-submission is the smart place to start

Here is the design decision I think gets undersold. PAT was piloted as a pre-submission tool for authors at STOC and ICML, two serious conferences, one theory-heavy and one machine-learning-heavy. That placement changes everything about the risk profile.

If you deploy AI on the reviewer’s side, you are making a verdict-shaping system. Every false positive becomes a paper unfairly dinged, every false negative becomes a bad paper waved through, and the author has no recourse against a machine they never see. That is the version everyone is rightly nervous about.

Pre-submission flips the incentive. The author runs PAT on their own work, sees the flagged issues, and decides what to do. A false positive costs the author a few minutes of “no, that’s actually fine.” A true positive saves them a desk reject or a humiliating erratum. The author stays in control, the human reviewers still do the judging, and the tool just raises the floor on what reaches them. That is the cognitive-burden argument in the paper: catch the dumb-but-costly errors early so referees can spend their attention on the parts that need human judgment.

I have argued before that the best place to put an imperfect AI checker is somewhere the human can easily override it and the cost of a miss is low. Pre-submission fits that exactly.

The catch the benchmark hides

Now the skeptical part. SPOT measures math errors, and a 34% recall bump there is a clean, checkable result. Math is the friendly case. A theorem is wrong or it is not, a derivation has a hole or it does not, and you can verify the catch.

The harder claims in the paper are softer. “Validating experiments,” “suggesting substantive improvements,” “identifying critical errors” in pilot deployments. Those are exactly the categories where current models are least reliable and where the failure modes are worst. An LLM that confidently tells you your ablation is missing a baseline when it is right there is annoying. An LLM that tells you your experimental setup is sound when it has a subtle leak is dangerous, and it is the kind of mistake these systems make.

The paper reports pilot results qualitatively, that PAT identified critical errors and suggested improvements at STOC and ICML, but the synthesizable hard evidence is the SPOT recall number. So the honest read is: strong signal on verifiable math errors, promising but unquantified on the messier review tasks. The danger is that the 34% headline gets generalized into “PAT reviews papers well,” when what we actually know is “PAT catches more math mistakes.”

There is also the obvious incentive loop nobody in the paper can fully solve. If authors run a Google tool to pass review, and reviewers eventually run AI to review, you get models grading models, with humans drifting toward rubber-stamping. The taxonomy’s higher rungs are where that risk lives. Google deserves credit for naming the ladder instead of pretending the top is the same as the bottom, but naming a slope is not the same as not sliding down it.

Why this still matters

Despite the caveats, I think this is the right direction. The review crisis is real, it is getting worse, and “just find more reviewers” is not a plan. A tool that catches your broken proof before three exhausted referees have to is a genuine good, even if it never touches a reviewer’s keyboard. The framing as collaboration levels, with control staying human at the assistive end, is more honest than most “AI for science” pitches I read.

The thing to watch is whether the soft claims firm up. Show me a benchmark for experimental-flaw recall, not just math, and show me false-positive rates authors actually tolerate. That is the difference between a useful pre-submission linter and another source of confident noise.

If you are a researcher, the move is simple: treat a tool like this as a self-check pass before you submit, the same way you run a linter or a spell-check, and never as a verdict. Run it, read the flags, fix the real ones, dismiss the wrong ones, and keep your own judgment as the final layer. The catch most people will miss is that the value is concentrated in the verifiable stuff, the math and the missing-baseline kind of error, and you should trust it least exactly where it sounds most impressive, on whether your ideas are any good. That part is still your job.