Visual pretraining is a bet against text extraction
A new arXiv paper argues that visual document pretraining can teach language intelligence better than plain text extraction, because equations, figures, and layout carry signal. The practical lesson is not “multimodal everything,” it is that lossy preprocessing may be quietly capping model quality.
The PDF was never just text
Most language model pipelines still treat visually rich material as a text source with annoying formatting attached. Strip the PDF. Extract the HTML. Flatten the paper, page, table, equation, and chart into a token stream. Train on that.
The arXiv paper Scalable Visual Pretraining for Language Intelligence challenges that default. The authors argue that many sources we call “text corpora” are really visual documents. Figures carry relationships. Typeset equations carry structure. Page layout carries hierarchy and intent. Tables carry alignment. Even whitespace can matter.
That sounds obvious if you have ever tried to understand a research paper from a bad OCR dump. It is less obvious at pretraining scale, where the industry has spent years optimizing around text as the clean substrate for intelligence.
The paper’s main claim is direct: unsupervised visual pretraining on the same underlying corpora consistently outperforms text-only pretraining across multiple backbones and benchmarks. The supplied abstract does not give effect sizes, so I would not overread the magnitude. But the direction matters. If the same content teaches better when seen as a visual artifact, then text extraction is not just a convenience layer. It is a lossy compression step.
Layout is data, not decoration
This lands because it reframes a boring infrastructure choice as a model capability choice.
Take equations. A plain text version can preserve characters while losing two-dimensional relationships. Take a scientific figure. A caption may describe it, but not fully. Take a webpage. Header position, sidebars, callouts, code blocks, and visual grouping tell you what the author meant users to read together. Text extraction often turns all of that into soup.
The paper is not saying language no longer matters. It is saying language intelligence may improve when the model learns from the forms where language actually lives. That is different from the usual multimodal pitch, where images are bolted onto a model so it can answer questions about photos. This is more basic: maybe the pretraining substrate for “language” should include visual documents by default.
I like this claim because it is operational. It points at a concrete bottleneck. Before arguing about bigger models, longer context, or more synthetic data, ask what signal got discarded on the way in.
There is also a cost question hiding here. Visual pretraining sounds heavier than text-only pretraining. Images of pages are larger than extracted text, and model architectures need to process spatial structure. The abstract frames visual pretraining as scalable and efficient, but without numbers in the provided material, that remains a claim to inspect in the full paper. Better benchmarks are nice. Better benchmarks per dollar are what change training recipes.
The next corpus fight is about representation
The last few years made data quality a first-class concern. Deduplication, filtering, licensing, synthetic data mixtures, and benchmark contamination all got attention. Representation quality deserves the same treatment.
A corpus is not just what documents you include. It is what version of those documents the model sees.
For builders, this matters even if you are not pretraining foundation models. RAG systems often make the same mistake at smaller scale. They parse PDFs into chunks, drop tables, scramble reading order, detach captions from figures, and then blame the retriever or generator when answers are bad. Document QA vendors know this pain well. The model cannot reason over structure it never received.
The practical move is to audit preprocessing as part of model evaluation. For any workflow involving PDFs, slides, forms, invoices, scientific papers, engineering docs, or websites, compare three pipelines: plain text extraction, layout-aware extraction, and image-based or multimodal ingestion. Use the same questions and measure failures by category, especially tables, equations, figures, and cross-page references. The catch most readers miss: visual pretraining is not a magic fix for messy documents. It is a reminder that “clean text” can be dirty data when the original meaning was visual.