The Cultural Cost of Building AI for 1.4 Billion People

Most of the AI conversation happens in English, about English, benchmarked in English. That is a problem when you are trying to serve a subcontinent that speaks hundreds of languages across a dozen writing systems, many of which barely register in the training data.

A new survey posted to arXiv (filed under both cs.AI and cs.CL) takes this head on. Titled “Rethinking Indic AI from a Lens of Cultural Heritage Preservation,” it traces the history of Indic NLP, lays out why Indian languages are genuinely hard to model, and proposes a research direction the authors call Culture Sensing. The framing I keep coming back to is theirs: AI as a double-edged sword. It can bring access and inclusion to a huge population, or it can homogenize worldviews and quietly erase the languages and perspectives that do not make it into the corpus.

That tension is not unique to India. But India is where it shows up at maximum resolution.

Why Indian languages break the usual playbook

The default recipe for a good language model is a lot of clean text plus scale. That recipe assumes the language behaves roughly like English: modest morphology, one dominant script, a big pile of digitized writing to learn from. Indian languages violate most of those assumptions at once.

The survey points to rich morphology, complex scripts and grammar rules, diglossia, and large dialectal variation as the core structural challenges. Each one hurts a model in a different way. Rich morphology means a single root can spawn dozens of surface forms, which wrecks tokenization and inflates the effective vocabulary. Complex scripts mean the same spoken word can be written several ways, and the byte-level tokenizers built for Latin text chop these scripts into inefficient fragments. Diglossia, where the formal written register differs sharply from everyday speech, means the text you can scrape does not match the language people actually use. And dialectal variation means “Hindi” or “Tamil” is not one target but a spread of them.

Stack these together and you understand why scale alone does not rescue low-resource Indian languages. You cannot brute-force your way past a data shortage that is also a representation shortage. The tokens are not just scarce, they are structurally awkward for architectures tuned on English and Chinese.

This is the part builders tend to underestimate. When a model does badly on Kannada or Bhojpuri, the reflex is “add more data.” The survey’s history of the field suggests the harder work has always been resource creation itself: building the corpora, the annotation, the evaluation sets that did not exist before someone chose to make them. That is slow, unglamorous, and mostly uncompensated by the incentives of frontier labs.

From patching gaps to Culture Sensing

The interesting move in the paper is that it does not stop at “we need more data and better tokenizers.” Those are real and the authors survey the progress: the methodological shifts over the years, the growing set of Indic foundation models trying to close resource and representation gaps.

But their proposed direction, Culture Sensing, is a different claim. They frame it around hermeneutic reasoning, which is the study of interpretation, of how meaning depends on context, tradition, and the reader. The argument is that fluency is not the same as cultural meaning. A model can produce grammatically perfect Tamil and still return output that is culturally hollow or wrong, because the connection between language and worldview in these traditions is tight. Kinship terms, honorifics, references to practice and ritual, all carry meaning that does not survive a literal translation.

Culture Sensing aims at two open problems the paper names directly: equitable performance across low-resource languages, and producing outputs that are culturally meaningful. The second is the ambitious one. Equitable performance is measurable, at least in principle. Cultural meaningfulness is much harder to pin down, and the paper is honest that this is a research direction, not a solved method.

I read this as a corrective to a benchmark culture that rewards the wrong thing. If your only metric is next-token accuracy or a translation score, a model that flattens every dialect into the dominant register will look great. It will also be doing exactly the homogenization the authors warn about. You get a scoreboard that goes up while the thing you actually cared about erodes.

What this means beyond India

The honest caveat: this is a survey and a proposal, not a shipped system with numbers. There is no benchmark here showing Culture Sensing beats a baseline, because the point is to reframe what the baseline should even measure. Treat it as agenda-setting rather than a result.

Still, the agenda travels. Every language community outside the English core faces some version of this. The specific failure modes differ. Arabic has its own diglossia problem, Indigenous languages of the Americas have extreme data scarcity, and so on. But the underlying question is the same everywhere: does an AI system serving a culture actually preserve that culture’s ways of meaning, or does it quietly assimilate them into whatever the training distribution favored?

That funnel image is the real risk. Not that AI ignores these languages, but that it “supports” them in a way that pulls everything toward a single flattened center. The paper’s contribution is naming that as a design failure rather than an acceptable rounding error.

Practitioner’s take

If you are building or fine-tuning anything for Indian language users, the operational lesson is to stop trusting a single aggregate score. Build evaluation sets that separate fluency from cultural correctness, and get native speakers, not just annotators, to judge whether outputs feel right in context. Test the dialects and the informal registers, not just the clean written form your corpus overrepresents, because that gap is where diglossia will burn you in production.

On the data side, invest in your tokenizer before you invest in more parameters. For morphologically rich scripts, a badly fragmented tokenizer silently taxes every downstream result, and no amount of scale fully repays it. Measure fertility (tokens per word) across your target languages and treat a bad number as a red flag.

The catch most readers will miss: Culture Sensing sounds like a nice-to-have you bolt on later. It is not. If your evaluation only measures fluency, you will optimize a model that gets more fluent and less meaningful at the same time, and your metrics will never tell you. Decide what culturally correct means, and measure it, before you scale. Otherwise you are building the exact homogenization machine the paper is warning about, and congratulating yourself on the benchmark while you do it.