The Tokenizer Tax on Non-English Users, and a Recipe to Cut It

There is a tax most people never see, and it falls hardest on people who do not read English. It lives in the tokenizer, the part of a language model that chops text into pieces before anything smart happens. If your language got a big slice of the pre-training corpus, your words become few tokens. If it did not, your words shatter into many. More tokens means more latency, more compute, more battery drain, and on metered APIs, a bigger bill. Same model, same prompt, different price depending on your alphabet.

A new paper titled “In-Place Tokenizer Expansion for Pre-trained LLMs” (posted across arXiv’s cs.AI, cs.CL, and cs.LG this week) tackles this directly. It comes from the team behind LFM2.5-8B-A1B, and it does something I have not seen packaged this cleanly before: it upgrades a model’s tokenizer after pre-training, in place, without wrecking the model that already learned to use the old one.

Why the tax exists

The authors frame it well. A tokenizer is frozen at the start of pre-training, so it reflects whatever the deployment priorities were at that moment. Vocabulary gets allocated in proportion to the corpus. When priorities shift later, say you decide Hindi and Vietnamese now matter, those languages get split into far more tokens per word because the vocabulary was never sized for them.

Cloud models can shrug this off. A giant model can afford a broad vocabulary because the embedding and LM-head matrices (the lookup tables that map tokens to vectors and back) are a small fraction of total parameters. On a compact on-device model, those same matrices are a material share of per-token decode bandwidth. So small models ship small vocabularies on purpose and accept fragmentation for anything outside a fixed language set. That is a defensible engineering choice. It is also, in practice, a choice about whose experience gets to be fast.

The size of the gap is not subtle. The authors report their source tokenizer splits Hindi and Vietnamese into roughly 2.4x and 2.6x more tokens than needed, and Thai up to 4x. A Thai user was effectively paying quadruple the token cost of an equivalent English sentence. That is the tax, measured.

The recipe, and why it is clever

The naive fix is to train a new tokenizer and start over. Expensive, and you throw away everything the model learned. The authors do the opposite. They keep the existing tokenizer and continue its BPE merges (byte pair encoding, the process that builds tokens by repeatedly merging frequent character pairs) on a multilingual corpus. This matters because of a compatibility property: most of the original tokens carry over unchanged as single tokens, and every new token decomposes exactly into old ones. Nothing about the old vocabulary breaks.

Then the initialization. Carried-over embedding rows get copied as-is. New tokens get initialized as the mean of their source sub-token embeddings. So the new token for a Hindi word starts life as the average of the fragments the model used to represent that word with. It is not random, it is a warm start pointed in roughly the right direction.

Finally, a two-stage adaptation. First train only the embeddings so the new rows settle without disturbing the rest of the network. Then continue full-model pre-training to recover source-checkpoint quality. The claim is that this sequence gets you back to the original model’s quality, not a degraded version of it.

They applied all of this to a continued pre-trained checkpoint of LFM2-8B-A1B, an 8B-parameter Mixture-of-Experts model, to help produce LFM2.5-8B-A1B with a 128K tokenizer. The payoff: combining the fewer-tokens-per-word reductions with the measured per-token cost of the larger vocabulary, they estimate a 2.2x to 3.7x per-character decode speedup for these languages across their reference devices. And they released the weights and the expanded tokenizer.

What I trust and what I would check

The honest part first: this is a single arXiv paper (mirrored across three categories, which is one submission, not three findings), from the team that ships the model in question. That is not disqualifying. Model producers are exactly who should publish this, since they control the tokenizer design. But it means the numbers are self-reported and the “recovers source-checkpoint quality” claim deserves scrutiny on real downstream benchmarks, not just perplexity.

Two things earn my trust. One, they report negative findings that shaped the recipe. Papers that only show the winning configuration are hiding the search. Papers that tell you what failed are describing an actual process. Two, the speedup number is presented as an estimate that combines token-count reduction with the measured cost of the bigger vocabulary. That is the correct way to reason about it, because a 128K vocabulary is not free. A bigger LM-head means more work per token. The net win only exists because the fewer-tokens effect outweighs the fatter-matrix cost. They did the subtraction instead of quoting only the flattering half.

What I would verify before betting on it: does the speedup hold on the specific devices you deploy to, or only their reference set? Does English quality survive untouched, or does it drift slightly when you rebalance the vocabulary? And how much continued pre-training compute did stage two actually cost? “Recovers quality” can hide a large training bill.

The bigger pattern

The deeper point is that tokenizer choices are deployment-priority choices frozen in time, and until now that freeze was permanent unless you retrained from scratch. If in-place expansion holds up, tokenizers become editable assets. You can add a language when a market opens, not at the model’s birth. For on-device AI specifically, where the vocabulary tax bites hardest, that is the difference between shipping a phone assistant that treats Thai as a second-class citizen and one that does not.

That reframes multilingual fairness from a data problem into an engineering-maintenance problem, which is a much better place for it to live.

For builders: if you serve non-English users on a small or on-device model, measure your tokens-per-word ratio per language before you do anything else. That single number tells you the tax you are charging today. If a supported language runs 2x or worse against English, you have a latency and cost problem hiding in plain sight, and now there is a documented path to fix it without a full retrain. The catch most people will miss: the speedup depends on your embedding and LM-head being a large share of decode bandwidth, which is true on compact models and false on large cloud ones. Do not copy this recipe onto a frontier-scale model expecting the same win. The whole method pays off precisely because the model is small.