Not a Translation Problem

The obvious way to build AI for India’s languages is to translate English datasets. The evidence on what that produces explains why Indian-language AI is a data-creation problem.

When a team at Google set out to build a question-answering benchmark covering eleven typologically diverse languages, they made an unusual decision. They would not translate anything.

The standard method for creating a multilingual dataset was, and is, to take an English one and translate it. It is fast and cheap and gives parallel data across languages. The authors of the benchmark, called TyDi QA, rejected it on principle. Translated text, they argued, is not the same as text written in a language. It carries the structure and concerns of the source. A question translated from English is a question an English speaker wanted answered, phrased the way English phrases it. So they had native speakers write questions from scratch in each language about things those speakers wanted to know.

The phenomenon they were avoiding has a name, translationese, and its effects on models are measurable. Artetxe and colleagues showed that translation leaves artefacts that models pick up on, and that evaluating on translated test sets can give a distorted picture of how well a model handles the language as written by its speakers.

For India, the point carries particular weight, because of how far the starting position is from English.

The shape of the problem

The 2011 Census of India recorded 121 languages spoken by ten thousand or more people. Twenty-two are listed in the Eighth Schedule of the Constitution, and together they are the mother tongue of 96.71% of the population. Behind those figures the census counted 270 identifiable mother tongues with ten thousand or more speakers, grouped from more than 19,500 raw responses.

Hindi, by the census’s definition, is the mother tongue of 43.63% of the population. Six further languages each account for more than 4%. The remainder is a long tail of languages with tens of millions of speakers each, more than most European languages have.

Speaker numbers bear little relation to digital resources. Joshi and colleagues at Microsoft Research classified the world’s languages into six tiers by the amount of labelled and unlabelled data available. English and a handful of others sat in the top tier. Hindi was placed a tier below, and most other Indian languages lower still, despite speaker populations in the tens of millions. A language can be among the twenty most spoken in the world and still be low-resource for AI.

What translation cannot supply

Content. A translated English corpus is about what English text is about. It will not contain the names of Indian places, institutions, foods, festivals, government schemes, kinship terms or legal concepts in the proportions in which they occur in real use. A model trained on it will be fluent on translated encyclopaedia content and thin on what users ask about.

Script. India’s scheduled languages are written in more than ten different scripts. Some languages use more than one. And a great deal of everyday digital text is not in the native script at all: it is typed in Latin characters, with spelling improvised by each writer. A user may write the same Hindi word three ways in one conversation. The AI4Bharat group built Aksharantar, a large transliteration resource, specifically because romanised input is how many people type. Translation of English produces clean native-script text and none of this.

Code-mixing. Multilingual speakers switch languages within a sentence, as a normal way of talking. Hindi and English, Tamil and English, Bengali and English, combined in a single utterance. Benchmarks built for code-mixed text, such as GLUECoS, found that multilingual models handled code-mixed text less well than text in a single language, and improved when exposed to code-mixed data. No monolingual source corpus contains it, so translating one cannot produce it.

Register and dialect. The formal standard of a language, as found in news and government documents, can differ considerably from how it is spoken in a given district. Translation systems output the standard. Users do not always write in it.

Context. Bhatt and colleagues examined what fairness evaluation would need to cover in India and showed that the relevant social categories differ from those in benchmarks developed in the West. Caste, religion and region carry a weight that an imported benchmark does not test. A dataset translated from English inherits the source’s assumptions about which distinctions matter.

Translation of the web by the web

There is a further difficulty. The alternative to translating datasets, collecting text from the web, is itself partly contaminated by translation.

As noted in the article on model collapse, Thompson and colleagues found that a large share of multi-way parallel content on the web appears to be machine-translated, and that this is concentrated in lower-resource languages. For a language with limited original web content, a scraped corpus may consist largely of machine output of uncertain quality. Training on it means learning from an earlier model’s errors.

Scraped data in major languages can be assumed to be mostly human-written, at least if it predates recent language models. For many Indian languages, that assumption is weaker.

What has been built

The response from Indian research institutions has been to create data, and the record over the past several years is substantial.

AI4Bharat, based at IIT Madras, has released a series of open resources. Samanantar assembled 49.7 million sentence pairs between English and eleven Indian languages. IndicTrans2 extended translation to all 22 scheduled languages, supported by a parallel corpus of 230 million sentence pairs, and the team created 644,000 manually translated sentence pairs as part of that work, along with what they describe as the first benchmark with parallel text across all 22. IndicCorp and related efforts gathered monolingual text.

The Government of India’s Bhashini mission, launched in 2022 under the National Language Translation Mission, funds language technology and data creation for Indian languages and runs a crowdsourcing initiative, Bhasha Daan, through which the public can contribute text and speech. The IndiaAI Mission, approved in 2024 with an outlay of a little over ten thousand crore rupees, includes a datasets platform among its components.

Two features of this work are instructive. Even translation models needed human translators: the 644,000 manually produced pairs were made because mined and machine-aligned data was not sufficient for training and was unsuitable for evaluation. And evaluation sets were built separately and carefully, because, as IndicGenBench and similar efforts recognise, without human-created test data in each language there is no way to know how well anything works.

The argument for translating anyway

Translation should not be dismissed. It has real uses and the pragmatic case is strong.

It is far cheaper than original collection. For instruction-following and reasoning data, where the skill being taught is largely independent of language, translated data transfers reasonably well. Large multilingual models show cross-lingual transfer: abilities learned in English partly carry over to other languages without any data in them. And machine translation for major Indian languages has improved to the point where its output is usable for many purposes.

A reasonable position is that translation is a good way to obtain volume for general capabilities and a poor way to obtain coverage of what is specific to a language and its speakers. The two are complementary.

The mistake is to treat a translated dataset as having solved the problem, and especially to evaluate on translated data. A model tested on translated English questions will be scored on its handling of translated English questions.

What original data collection involves

Creating data in a language, as distinct from into it, has practical requirements.

Native speakers as authors. Prompts, questions and documents written by people who use the language daily, about subjects they choose.

Regional and social spread. Hindi in Bihar is different from Hindi in Rajasthan. A language’s data should be collected across the places and communities where it is spoken.

Real input forms. Native script, romanised text and code-mixed text, in proportions that reflect how people type.

Domain coverage. Agriculture, health, finance, law and government services, in the vocabulary used locally.

Native-speaker evaluation. Test sets written and verified by speakers, never translated, with the same care this series has argued for throughout.

Annotators who share the context. Judging whether a response is appropriate, offensive or accurate in Marathi requires a Marathi speaker who knows the setting.

The last point connects India’s linguistic situation to its workforce. The country has large numbers of educated native speakers of every scheduled language, many of them multilingual. That is the input that cannot be imported or synthesised.

What this changes