RAG For Table-Heavy Reports Needs Search You Can Audit
Dense retrieval breaks on table-heavy reports because numbers lose headers, units, and fiscal years. The arXiv paper “Beyond Top-K” argues for deterministic document operations over opaque nearest-neighbor chunks, with useful evidence and one important limit: lexical search may be doing much of the work.
TL;DR: For financial statements, audit reports, and regulatory filings, the safer RAG pattern may be less “embed everything” and more “give the model auditable document operations.”
What actually breaks in top-k retrieval?
The primary source here is the arXiv paper “Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations,” listed under cs.AI and cs.CL. Its claim is narrow, which is why I like it.
This is not “vectors are dead.” It is: top-k dense retrieval is a bad default for long, table-heavy documents where numbers depend on nearby structure.
The paper uses a 780-page government financial report as its test case. In that report, 86.8% of content lines are table rows. That matters because tables are not prose. A number can look identical to thousands of other numbers. Its meaning may depend on a unit header, a fiscal-year header, a section title, or row context that sits far away in text order.
The nastiest example is units. The paper reports that a figure inherits its unit from a header a median of 13 lines above it. If a chunk boundary splits the number from “lakh” or “crore,” the system can be wrong by two orders of magnitude. That is not a small retrieval miss. That is the kind of error that makes an answer unusable.
The researchers even built a table-aware chunker as a stronger baseline. It fixed the unit problem, but still left 27% to 30% of numeric chunks without a fiscal-year header across every chunk size they tried. So the problem is not just lazy chunking. It is that the document’s meaning lives in layout and hierarchy, while the retrieval system sees chopped text competing in one embedding space.
Is the win from agents, or from better search?
The paper proposes READ, short for Reliable Embedding-free Agentic Document-search. The agent gets three deterministic operations over the raw document: normalized lexical search, structural navigation, and bounded span reads. These are exposed over the Model Context Protocol, so the model’s search path can be replayed as an audit trail.
That auditability is the practical shift. A dense retriever says, roughly, “these chunks were close.” READ can show the steps: searched this phrase, moved to this section, read this bounded span. For regulated or finance-adjacent work, that difference matters.
The reported numbers are strong. On 51 verified questions, READ answered 58.8% correctly. Dense retrieval got 15.7%. A tuned dense setup reached 35.3%, still 23.5 points behind READ. An agent with the same loop but only a top-k tool reached 27.5%, which suggests the gain is coming from the document interface, not just from letting a model iterate.
But the paper is careful about one limit, and builders should be too. BM25 was statistically indistinguishable from READ. That means the evidence separates embedding-based retrieval from embedding-free retrieval. It does not prove that “agentic” is the magic ingredient. Plain lexical search remains annoyingly hard to beat when the task depends on exact terms, labels, and document-local evidence.
That is a useful anti-hype result. Sometimes the “agent” is less important than the tools you put in its hands.
What should builders change in production RAG?
If your documents are prose-heavy, dense retrieval can still be a good default. Policies, memos, docs, emails, support articles. Fine.
If your documents are table-heavy, numeric, or audit-sensitive, I would stop treating embeddings as the only retrieval layer. Add deterministic search. Preserve page, table, row, column, and header structure. Let the model read bounded spans from the original document. Log every operation. Make citations point to locations that a human can inspect.
The more important design move is to test retrieval before testing generation. Ask whether the correct unit, year, row header, and table title are present in the context window. If they are missing, no model prompt will reliably save you.
Practitioner’s Take: Build a small eval from your own worst documents, not a generic RAG benchmark. Pick 30 to 50 questions where the answer depends on units, dates, headers, footnotes, or table structure. Compare dense top-k, BM25, and a structured read tool. The catch most teams miss: the model may look smart while answering from context that is already corrupted. Fix the document interface first.