ScholarCatalyst: When Agentic Search Can't Beat a Plain Embedding

TL;DR: On ScholarCatalyst, a benchmark where AI has to retrieve the prior papers that actually advanced a real research project, agentic search scored 0.42 Recall@20 versus 0.48 for a plain embedding retriever, which means wrapping a search tool in an agent loop did not buy you better taste about which ideas matter.

That result should bother anyone building research agents. The pitch for agentic retrieval is that reasoning on top of search beats search alone. Here it did not. The agent called the same retriever as a tool and still came out behind.

What is ScholarCatalyst actually measuring?

The benchmark comes from the paper ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research, posted to arXiv under both cs.AI and cs.CL. The setup is the clever part. Most retrieval benchmarks test whether a system can find documents that match a query. This one tests something harder: given a half-formed research question, can you find the prior work that would actually push the project forward.

The labels are the reason to care. The authors recruited 184 lead authors of 207 recent computer science papers and had them mark which candidate papers did or could have advanced their own projects, each with a written rationale. These are not crowd workers guessing at relevance. They are the people who finished the project and know firsthand which earlier idea mattered. An automated pipeline made that annotation scalable, which is how you get judgments on 207 projects instead of a handful.

The retrieval task adds a constraint that most benchmarks skip: you can only retrieve from the literature available when the project began. No peeking at papers published after the idea started. That turns it from a lookup problem into something closer to the actual skill, which is sensing which buried prior idea a new problem needs.

Why did the agent not beat the embedding?

Here is the number that makes the paper worth reading. Agentic search scored 0.42 Recall@20. A plain embedding retriever scored 0.48. The agent called that same retriever as a tool and still lost ground.

That is the opposite of the usual story. We have gotten used to the idea that adding a reasoning loop on top of a retriever improves results: the model reads what came back, reformulates the query, searches again, filters. On this task that machinery either added noise or chased the wrong signal. The agent had the right tool in its hands and used it worse than the tool used alone.

Even the strongest configuration tops out low. An agent built on Claude Fable 5.1, which the authors note may have seen the completed papers during training, reaches only 0.51 R@20. Think about that. A model that might have memorized the answers, given twenty shots to surface the right prior work, catches barely half of it. The contamination risk cuts the wrong way for the optimists: even with a possible leak, the ceiling stayed low.

I want to be careful here. Claude Fable 5.1 is the model name as given in the source, and I have no first-party documentation on it beyond what the paper states, so treat the training-contamination claim as the authors’ caveat rather than confirmed fact. What is not in dispute is the gap between agent and embedding, because both numbers come from the same benchmark run.

What does this tell us about taste as a capability?

The framing the authors use is “expert intuition for searching broad corpora.” I would call it taste. A good researcher does not just match keywords. They feel which neighboring field holds the trick they need, often before they can articulate why. That is a learned sense built from years of reading, and it is exactly the thing these systems do not have yet.

Current retrieval, agentic or not, is good at finding what is near your query in meaning space. It is weak at the move that matters most in research, which is the non-obvious jump: the paper from an adjacent discipline, the old technique repurposed, the method nobody connected to your problem yet. Embedding similarity rewards closeness. Great prior-work discovery often rewards the opposite.

The authors argue this points to a need for new training recipes rather than better prompting or bigger context windows. I think that is the honest read. You cannot prompt your way to a skill the base model never learned. If the signal for “this distant paper is the key” is not in the training objective, no agent scaffold recovers it. The scaffold just reorganizes what the model already knows.

What should builders take from a benchmark they will never run?

Most people reading this are not building scientific literature agents. The lesson still lands. Any time you bolt an agent loop onto a retriever and assume it beats the retriever, you are making a bet that ScholarCatalyst just lost. Measure it. Run the plain baseline. If your agent does not clear it, the loop is costing you tokens and latency to make results worse.

This matters beyond research. Anyone building retrieval over a corpus, legal discovery, support knowledge bases, internal docs, content archives for SEO, lives in the same space. Finding the obviously relevant document is close to solved. Finding the non-obvious one that actually answers the question is not, and that is usually the one that earns its keep. (Disclosure for the SEO angle: I run Lucky Domains, which does white-hat SEO and search visibility work, so I watch this gap closely.)

The practitioner move is to treat agentic retrieval as a hypothesis, not a default. Start with embedding retrieval, measure recall at whatever cutoff your product uses, then add the agent only if it beats that number on your own labeled set. Build that labeled set the way ScholarCatalyst did: get the people who know the right answer to mark it, with rationales, not proxy relevance from a judge model. The catch most readers will miss is that the 0.42 versus 0.48 result is not a knock on agents in general. It is evidence that your scaffolding only helps when the base model already has the taste the task demands, and for discovering non-obvious prior work, it does not yet. Spend your effort on the data and the eval before you spend it on the loop.