Back-translation beats more data in low-resource translation
An arXiv paper argues that old-school back-translation is still a strong move for low-resource machine translation with LLMs, especially when you have monolingual text but very little parallel data.
TL;DR: If you are building translation for a low-resource language, monolingual text plus back-translation may beat both continued pretraining and simply hunting for more parallel data.
What does back-translation change for low-resource LLM translation?
The primary source is the arXiv paper “Back-Translation Still Matters: Leveraging Monolingual Data for Low-Resource Machine Translation with LLMs,” listed under cs.AI and cs.CL. Its core claim is refreshingly practical: when parallel data is scarce but monolingual text exists, do not default to continued pretraining. Generate translation pairs from the monolingual text, then fine-tune on those synthetic pairs while still mixing in the real parallel examples.
That is an old machine translation trick. The interesting part is that it still works in the LLM era.
The setup is straightforward. Start with a fine-tuned LLM. Feed it monolingual sentences in one language. Have it translate them into the other language. Now you have synthetic parallel data. Train in the reverse direction too. Interleave updates on the small human-created parallel set so the model does not drift too far into its own synthetic mistakes.
This matters because the default LLM instinct is often, “add more pretraining.” That is expensive, vague, and not always the best use of the data. The paper reports that with only 1,000 parallel pairs on Odia and Swahili, back-translation beat continued pretraining into the low-resource language by 7.7 to 16.4 XCOMET-XXL points in-domain. That is not a tiny gain hidden behind vibes.
Why not just collect more parallel data?
Because parallel data is the bottleneck.
For many low-resource languages, clean sentence-aligned pairs are scarce, expensive, politically uneven, or locked inside institutions. Monolingual text is not always abundant either, but it is often easier to find: books, websites, public documents, educational material, local media, religious texts, community archives.
The paper’s strongest operator-friendly result is that beam-decoded intermediate translations beat a model trained with 10 times more parallel data. Read that carefully. It does not mean data collection is pointless. It means the format and use of the data matter. A smaller amount of real parallel data can become much more useful when it anchors a larger synthetic loop.
Across five low-resource languages, with Hindi used as a high-resource control, and two 3B backbones, the paper reports 16 to 173% XCOMET-XXL improvement over parallel-only fine-tuning for translation into the low-resource language. Blind human evaluation on Odia and Sanskrit agreed with the automatic metrics.
That last point is important. Translation metrics can lie, especially across language families and domains. Human evaluation is not magic either, but it helps keep the result from being only a benchmark artifact.
What is the catch?
Synthetic data can poison itself. If the model’s first translations are bad, the next training round can preserve those errors with confidence. The paper addresses this partly by interleaving real parallel updates and by testing quality, source, and amount of monolingual text. It also reports gains from retrieved in-context examples from the parallel data.
That suggests a useful recipe, not a universal law.
Use the best small parallel set you can get. Keep it in the loop. Use monolingual data that matches the domain you care about. Generate translations with stronger decoding when quality matters. Evaluate with both automatic metrics and humans who know the language. Watch for fluency that hides wrong meaning, especially with names, numbers, honorifics, cultural terms, and domain jargon.
My take: this is a good reminder that “LLM-native” does not always mean “brand-new technique.” Sometimes the winning move is an older method with a better base model and tighter evaluation.
For builders, I would try this before funding a large annotation push. Take 1,000 to 5,000 strong parallel pairs, add a carefully cleaned monolingual corpus, generate back-translations, then fine-tune with real pairs mixed into every stage. The catch most teams miss is domain fit. A model improved on general text can still fail on medical, legal, government, or education content if the monolingual pool does not look like the job.