Auxiliary views explain why diverse pre-training data works
A paper on LLM knowledge acquisition argues that models learn facts better when training data includes multiple reformulations, not just repeated documents. The useful lesson for builders is narrower than synthetic data hype suggests.
TL;DR: LLMs do not just need to see facts again, they often learn them better when the same knowledge appears through different reformulations, contexts, and supporting concepts.
What is an “auxiliary view” in pre-training?
The primary source here is the arXiv cs.AI and cs.CL paper titled “Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views.” No author list or arXiv ID was included in the supplied material, so I’m not going to invent one.
The paper’s core claim is simple and useful: pre-training data helps models acquire knowledge not only by repeating facts, but by presenting those facts through auxiliary views. That means reformulations of the same underlying knowledge.
Not just exact duplicates. Not just “Paris is the capital of France” 1,000 times. More like the same fact appearing in different sentences, contexts, explanations, nearby facts, or conceptual frames.
That matters because a lot of AI data talk collapses into one crude variable: more tokens. This paper points at a more interesting variable: how many different ways the model gets to compress the same underlying knowledge.
The controlled experiments reported in “Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views” try to isolate that effect. The paper confirms that repetition is still necessary for acquisition. A model generally needs multiple exposures. But repetition alone is not the whole story. When the token budget is fixed, shifting some tokens away from repeated documents and toward auxiliary views improves learning, including factual recall.
That last part is the hook. Even for recall, where you might expect exact repetition to dominate, varied representations can win.
Does this mean synthetic paraphrases are always better than repeating data?
No. This is where the hype version would get ahead of the evidence.
The paper reports that paraphrasing helps only at smaller batch sizes. That is a real caveat, not a footnote. Batch size changes the training dynamics. If you are reading this as “just generate a bunch of paraphrases and call it pre-training data,” you are probably over-reading it.
There is another interesting finding: the effectiveness of auxiliary views was not contingent on the strength of the teacher model that generated them. That cuts against one common assumption in synthetic data pipelines, where teams often treat the biggest teacher model as automatically better. The paper’s result suggests the value may come less from oracle-level generation and more from useful variation in representation.
Still, “variation” is not the same as noise. If the auxiliary view is wrong, misleading, or just a low-effort rewrite with the same surface pattern, it may not help. The supplied abstract does not quantify how much improvement came from each kind of view, and it does not prove the result transfers cleanly to every frontier-scale pre-training run.
The paper also separates contextual and foundational forms of knowledge. That distinction feels important. A model with a prior knowledge gap may need surrounding context, but it may also need prerequisite facts that make the target fact easier to encode. Humans work this way too. You do not learn a technical fact well if you lack the concepts it sits on top of.
What should builders take from this?
For training teams, the practical takeaway is not “deduplicate less” or “deduplicate more.” It is: deduplicate with intent. Exact duplicates are not the same as distinct explanations, examples, references, and prerequisite material. A web corpus full of naturally varied mentions may teach better than a cleaner but flatter dataset.
For fine-tuning and domain adaptation, this points to a better recipe than dumping one canonical manual into the trainer 20 times. If you want a model to learn your product, API, policy, or domain vocabulary, create multiple legitimate views: reference docs, task examples, troubleshooting notes, Q&A, migration guides, edge cases, and short conceptual explainers. Same knowledge. Different angles.
For RAG builders, this is also a reminder that retrieval quality is not only about finding the one perfect chunk. Sometimes the model benefits from seeing the answer chunk plus adjacent framing: definition, example, prerequisite, and constraint. Context design can mimic auxiliary views at inference time, even if you are not training the model.
I would try this first on a small eval-backed workflow. Pick 50 to 200 facts or procedures your model misses. For half, add exact repetitions. For the other half, add varied auxiliary material: paraphrase, example, prerequisite, and common failure mode. Keep the token budget roughly comparable. Then test recall and task performance separately. The catch most readers miss: diversity only helps when it preserves the underlying truth. Variation is useful. Drift is expensive.