The Bengali Gap: Why Language Models Fail Before They Train

TL;DR: A model can be biased against a language before a single training step runs, because the corpora, tokenizers, benchmarks, and deployment assumptions were all built without that language in mind, and the paper “Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages” shows this compounding failure in detail for Bengali.

Most conversations about AI fairness start at the model. Did it hallucinate, did it stereotype, did it refuse the wrong thing. That framing assumes the model is where the problem lives. The paper “Structural Silence,” posted to arXiv under both cs.AI and cs.CL, makes a sharper argument: for speakers of underrepresented languages, the disadvantage is baked into the infrastructure long before anyone picks a loss function. By the time you’re evaluating a chatbot’s Bengali, the deck was stacked three layers down.

Bengali is a useful test case precisely because it isn’t obscure. It’s one of the most spoken languages on the planet, representing nearly 4% of the global population. If a language with that many speakers gets shortchanged, the problem clearly isn’t about how many people use it. It’s about who built the pipeline.

What are the four failures the paper identifies?

The authors describe four “interlocking” failures, and the word matters. These aren’t four separate bugs you could patch one at a time. They multiply.

First, the web presence gap. Bengali accounts for less than 0.5% of global web content while representing close to 4% of humanity. That’s an eight-to-one mismatch between how many people speak it and how much of the internet is written in it. Since web crawls are the raw material for most large models, a language that’s underrepresented online is underrepresented in the model’s world before anyone makes a single design choice.

Second, the training-token deficit. In major multilingual corpora, the paper reports a 67:1 ratio between English and Bengali tokens. So even after a lab makes an explicit effort to be “multilingual,” the actual budget a model spends learning Bengali is a rounding error next to English.

Third, and this is the one most people miss, the tokenization penalty. Bengali uses an alphasyllabary script, and the paper notes this drives higher “token fertility,” meaning the same amount of text gets chopped into more tokens than it would in English.

Higher token fertility compounds everything above it. If Bengali already has fewer tokens in the corpus, and each unit of Bengali meaning costs more tokens to represent, then the effective learning signal shrinks twice. It also makes Bengali more expensive to run at inference, because you pay per token, and it eats context windows faster. A user asking the same question in Bengali and English is not getting the same product at the same price.

Fourth, connectivity exclusion. The paper cites individual internet penetration at 36.5% in rural areas versus 71.4% in urban ones. So even a perfectly capable model doesn’t reach the people who’d benefit most, because the deployment assumption, always-online, always-connected, quietly excludes them.

Why does “structural” change how you fix it?

Here’s the load-bearing claim, and I think it’s right: the authors argue dataset scarcity should be understood as a structural barrier, not an isolated technical limitation.

That reframing has teeth. If Bengali data scarcity is a technical problem, the fix is “collect more data” and you’re done thinking about it. If it’s structural, the scarcity is a downstream symptom of resource-allocation decisions, institutional priorities, and design defaults that never centered these languages in the first place. You can scrape all the Bengali you want and still ship a tokenizer tuned for English, benchmarks written in English, and an architecture that assumes broadband.

I’d push this one step further than the paper. Structural problems reproduce themselves. Low web presence means low training tokens, which means weaker models, which means fewer people build Bengali products, which means less Bengali content gets created, which keeps web presence low. It’s a loop. And every “multilingual” model that ships with an eight-to-one representation gap quietly ratifies the loop as normal.

What can a builder actually do with this?

This is where the paper offers something concrete rather than just a diagnosis. It treats offline-first design as an equity-oriented infrastructure strategy, not a degraded fallback. That’s a genuine shift. Most teams treat offline as the sad mode you get when the network drops. The paper says for a rural Bengali student it’s the primary mode, and designing for it first is the equitable choice.

I want to be honest about what the sources do and don’t give us here. The paper names the four failures and points toward directions for linguistics and AI research to reduce them, but the abstract available to me stops short of a fully specified engineering recipe. So treat the offline-first framing as the paper’s argued position, not a benchmarked, deployed system with numbers attached. The evidence in the abstract is the diagnosis. The prescription is a direction.

Does this generalize beyond Bengali?

The specific figures are Bengali. The mechanism is not. Any language with a non-Latin script, thin web presence, and a rural speaker base will hit some version of the same four walls. Amharic, Sinhala, many South Asian and African languages, share the alphasyllabary or complex-script tokenization penalty. The web presence gap and connectivity exclusion apply to hundreds of languages.

What Bengali proves is that scale of speakers doesn’t save you. If a top-tier language by population still loses to infrastructure defaults, then the smaller languages behind it aren’t a little worse off. They’re structurally invisible.

The practitioner’s take

If you’re building anything that claims to serve non-English users, audit the four layers before you audit the model. Run your own tokenizer against target-language text and measure token fertility directly: if the same sentence costs two or three times the tokens it does in English, you’ve found a cost-and-quality tax your users pay and you probably haven’t priced. Check what fraction of your actual training or fine-tuning data is the target language, not what the corpus card claims is “supported.” And design the offline path as a first-class feature, not a graceful-degradation afterthought, because for a real chunk of your intended users it’s the only path.

The catch most readers miss: fixing the model is the easy 20%. The hard 80% is the corpus, the tokenizer, the benchmark, and the connectivity assumption, and those get decided by people who usually aren’t in the room when the “add multilingual support” ticket gets written. “Structural Silence” is worth reading precisely because it moves the fairness conversation upstream, to where the decisions actually get made.