The multilingual loophole in LLM unlearning
A 174-language benchmark shows that removing a fact from a model in one language can leave it reachable through another. The useful idea is not brute-force forgetting everywhere, but coverage-aware language selection when unlearning budgets are limited.
TL;DR: LLM unlearning needs multilingual testing, because a fact that looks erased in one language can still be recovered by asking differently in another.
What is the cross-lingual unlearning loophole?
The primary source here is the arXiv cs.AI/cs.CL paper, “Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning.”
The core finding is simple and annoying: unlearning a fact in one language does not mean the model has forgotten it everywhere. Change the query language, or ask for the answer in another language, and the same supposedly forgotten knowledge may come back.
That matters because most practical unlearning workflows are already full of shortcuts. A team removes a fact. They test the most obvious prompts. They test in English, maybe a few other major languages. The eval passes. The ticket closes.
But the model did not necessarily forget the knowledge. It may have learned not to express it under a narrow set of linguistic conditions.
The paper names this as a cross-lingual loophole. I like the framing because it moves unlearning away from a vibes-based “did the model refuse?” test and toward an access problem: can a user still reach the target knowledge through a different expression of the same request?
That is the right mental model. Unlearning is not a delete key. It is a behavior change over a huge surface area.
Why not just unlearn in every language?
The obvious fix is brute force: apply the forget supervision across all languages.
The paper argues that this is neither scalable nor desirable. Scalable is easy to understand. The benchmark spans 174 language-script pairs and 25 atomic paraphrase types. Multiply that across many facts, many model families, and repeated safety or privacy updates, and “just test everything” turns into a permanent tax.
The more interesting point is damage. Broad unlearning can hurt unrelated model capabilities. If the goal is to remove access to a specific fact, blasting supervision across every language and phrasing may erase more than intended.
So the paper introduces “language budgeted multilingual unlearning.” The goal is to choose a subset of languages that gives the best cross-lingual erasure when you cannot afford, or do not want, to supervise every language directly.
That sounds obvious until you hit the composition problem. The paper reports that naively picking strong individual source languages does not reliably produce a strong set. In other words, the best one-at-a-time choices may overlap in what they cover, leaving gaps elsewhere.
What does COVER actually change?
The paper proposes COVER, a method that selects source languages to maximize predicted coverage of languages that receive no direct forget supervision. At deployment, COVER requires benign calibration data and access to the frozen model.
That last part is practical. It means the method is not framed around secret training internals or privileged access to the full original training process. It is aimed at choosing better unlearning inputs under a real constraint.
Across three model families and two disjoint forget sets, COVER reduced mean held-out residual access by 7.8% to 27.3% relative to uniform source selection. The paper also reports that the gains extend beyond synthetic benchmarks to real news documents in low-resource language settings using human translated data from the Low Resource Languages for Emergent Incidents, or LORELEI, corpus.
I would not read that as “multilingual unlearning is solved.” The result is narrower and more useful than that. COVER reduces residual access compared with a baseline. It does not prove total erasure. It also depends on the quality of the benchmark, the calibration setup, and how well the chosen language set matches the ways real users probe the model.
Still, this is the shape serious unlearning work needs to take. Not just “remove this from the model,” but “measure the remaining access paths, especially the weird ones.”
If you are building with models in regulated, multilingual, or safety-sensitive contexts, treat unlearning as an eval design problem first. Pick a target fact set, test across query language and requested answer language, include paraphrases, and track residual access instead of relying on a few refusal examples. The catch most teams miss: the language you use to remove knowledge may not be the language that proves it is gone.