Teaching Models Formal Logic Before Words: What Logic-PPT Actually Shows

TL;DR: Warming up a language model on formal logic derivations before feeding it natural text let it hit 80% on linguistic tasks using 36 billion fewer tokens, and the same trick reshaped the model’s internals so it survives being pruned to about 33% sparsity without losing ground.

The paper is “Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility,” posted to arXiv across cs.AI, cs.CL, and cs.LG. The core claim is simple to state and harder to dismiss than most pretraining tricks: start a model on formal logical derivations, then switch to normal text, and it learns language faster and compresses better afterward. What makes this worth reading is not the token savings alone. It is the mechanistic story about why the savings happen.

What is pre-pretraining and why bother?

Pre-pretraining is the step before the step. You take a fresh model and run it on some structured, non-natural data first, then do the real pretraining on natural language. The idea has been around: people have tried Dyck languages (nested brackets) and procedural algorithm traces as the warmup. The pitch is that structured data plants useful inductive biases, so when the model finally sees text, it already has scaffolding for hierarchy and dependency.

The authors’ complaint about prior work is specific. Dyck and algorithmic tasks lean on narrow primitives. Balanced brackets teach nesting, but they do not teach variable binding, quantifier scope, or how a predicate relates to its arguments across a long span. Natural language needs all of those at once. So the warmup was too thin to transfer much.

Their answer is Logic-PPT: use formal derivations as the warmup task. A formal derivation is a proof-like chain where you bind variables, connect quantifiers, track relational dependencies, and compose predicate-argument structures over a long context. That is a much closer rehearsal of what language actually demands. The bet is that if you make a model good at the abstract machinery first, the words come cheaper later.

Does the token saving actually hold up?

The headline result is 80% accuracy on linguistic tasks reached with 36B fewer tokens than a standard random initialization. Two things make me take this more seriously than the usual pretraining-trick claim.

First, the scale. A lot of pre-pretraining studies run at small token budgets where any effect could be noise or a quirk that vanishes once you train for real. This one runs to a 100B-token regime. That is not frontier scale, but it is large enough that a 36B-token head start is a real slice of the training run, not a rounding error. If you are spending compute to train a model, cutting a third of your tokens to reach a target is money.

Second, they compared against other pre-pretraining baselines, not just against a cold start. Logic-PPT beat the alternatives, which suggests the gain comes from the richness of formal derivations specifically, not just from “any structured warmup helps.” That distinction matters. If Dyck and algorithm traces gave you the same lift, the logic framing would be marketing. The claim is that they do not.

The caveat I would flag: “linguistic tasks” is doing quiet work in that sentence. The abstract does not enumerate which benchmarks, and 80% on a curated linguistic evaluation is not the same as downstream performance on the messy tasks people actually deploy models for. Faster acquisition of structural competence is plausible. Whether it translates to better real-world generation at the end of a full training run is a separate question the abstract does not answer. Read the tables before you budget around this.

Why does logic warmup make a model easier to prune?

This is the part I find genuinely interesting, and it is where the paper goes beyond “here is a trick that works.”

The authors report that formal derivations cause persistent structural reorganization inside the model. Specifically, the representation space becomes lower-rank and spectrally concentrated. In plain terms: instead of spreading information across many roughly-equal directions, the model packs it into fewer dominant ones. The internal geometry gets tidier.

That tidiness has a payoff they measured. The reorganized model prunes better. They report matching the dense baseline’s performance at roughly 33% sparsity, meaning you can zero out about a third of the weights and lose nothing. A lower-rank representation is, almost by definition, one where a lot of the parameters are redundant, so pruning finds slack to cut without hurting the parts that matter.

This connects two things people usually treat separately: what you train on and how compressible the result is. The usual pruning story is post-hoc. Train a dense model, then find weights to remove. Logic-PPT suggests you can shape compressibility upstream by choosing a warmup that induces a friendlier geometry. The warmup and the pruning are the same lever seen from two ends.

I want to be careful not to oversell the causal chain. The abstract asserts formal derivations “induce” the low-rank geometry and that the geometry “enables” better pruning. Those are correlational-sounding claims dressed as mechanism, and a single paper’s spectral analysis is suggestive, not settled. It is a clean hypothesis with measured support, which is more than most efficiency papers offer, but it is one result on one training setup.

Where does this fit in the efficiency conversation?

The efficiency world has been chasing three levers: better data, better architectures, and better compression after the fact. Logic-PPT is interesting because it touches data and compression at once. You spend a bit of compute on a synthetic logic warmup, and you get both faster language learning and a model that tolerates pruning. If both effects survive replication at larger scale, the warmup pays for itself twice.

The open questions are the ones that always decide whether a lab result becomes standard practice. How much does the logic warmup itself cost in compute, and does it net out positive once you count it? Does the effect hold at 1T tokens and frontier model sizes, or does it wash out as the main training run dominates? Do the low-rank representations help or hurt on tasks that need many fine-grained distinctions, where you might actually want more spread-out directions, not fewer? None of that is answered here, and none of it should be, from one paper.

Ken’s Practitioner’s Take: if you train or fine-tune models from scratch, this is worth a small experiment before it is worth a strategy. Generate a corpus of formal derivations (there are theorem-prover and synthetic-logic pipelines that can do this), run a modest logic warmup, then your normal pretraining, and measure two things: tokens-to-target on your eval, and how far you can prune the finished model before quality drops. The catch most readers will miss is that the pruning result may be the more valuable half. Everyone chasing the token savings will overlook that you might be able to ship a smaller, cheaper model at inference time for free, just by changing what the model saw in its first few billion tokens. If you are not training from scratch, this does nothing for you today. It is a recipe for model builders, not a prompt for users.