The Voice Gap

Speech is how most people in India will reach AI. It is also the modality in which the distance between available data and real use is greatest.

In 2020, researchers at Stanford tested the speech-recognition systems of five large technology companies on recordings of interviews with Black and white speakers in the United States. The systems had been trained on very large quantities of audio and were in commercial use.

For white speakers, the systems misrecognised on average 19 words in every hundred. For Black speakers, 35.

The cause was not the language, which was English in both cases. Analysis pointed to the acoustic models: the systems had been trained on speech that sounded more like one group than the other. The difference in accuracy was a difference in training data, expressed as a difference in service.

That study concerns one country and two groups. Extend the same logic to a country with twenty-two scheduled languages, hundreds of dialects and a very large population for whom speaking to a device is more natural than typing, and the scale of the data question becomes clear.

Accuracy follows hours

The relationship between training audio and recognition accuracy is among the better-established facts in the field.

OpenAI’s Whisper model was trained on 680,000 hours of audio collected from the web. About 117,000 of those hours covered 96 languages other than English. The paper reports a strong relationship between the amount of training data for a language and the model’s error rate in it: languages with more hours were recognised better, across the range. The authors note that many of the languages with poor performance simply had very little data.

Divide 117,000 hours among 96 languages, unevenly, and many Indian languages are left with a small share. And hours alone are not the whole story. Audio collected from the web is mostly prepared speech: broadcasts, audiobooks, lectures, subtitled video. It under-represents ordinary people talking in ordinary settings.

What Indian speech data needs to cover

Language. Each of the scheduled languages, and the many widely spoken languages outside the schedule.

Region and accent. Speech varies by district. The Svarah benchmark was created to test English recognition on Indian accents, with speakers from across the country, because models accurate on American and British English were noticeably less so on Indian English.

Speaker. Age, gender, education and rural or urban background all affect speech. A dataset recorded from university students in two cities describes those students.

Style. Reading a prepared sentence is acoustically and linguistically different from speaking spontaneously. Conversation, with interruptions, hesitations and overlap, is different again.

Language-mixing. Speakers switch between languages mid-sentence. Sitaram and colleagues’ survey of code-switched speech and language processing describes how this breaks the assumption, built into most systems, that an utterance is in one language. A shared task on multilingual and code-switched recognition for Indian languages was organised at a leading speech conference in 2021 because the problem was recognised as distinct.

Conditions. Telephone audio, low-cost handsets, background noise from streets, markets and households.

Vocabulary. Names of people, places, crops, medicines, schemes and products.

Each of these is a dimension along which a dataset can have coverage or lack it, in the sense set out in the article on data quality. A model fails along whichever dimension its data did not span.

How the datasets have evolved

The sequence of open speech datasets from Indian research groups shows a clear progression in what was understood to matter.

Kathbath provided 1,684 hours of labelled speech across 12 languages from 1,218 contributors in 203 districts. It was read speech: contributors read sentences shown to them. This gives clean, well-transcribed audio and is relatively efficient to collect.

Shrutilipi took a different route, mining about 6,400 hours across 12 languages from the archives of All India Radio by aligning broadcast audio with its transcripts. This yielded volume at low cost, in the formal register of news bulletins.

IndicVoices changed the method. It collected 7,348 hours from 16,237 speakers across 145 districts and all 22 scheduled languages. Only 9% is read speech. Seventy-four per cent is extempore, where speakers responded in their own words to prompts about their lives and interests, and 17% is conversation. The authors describe a collection process involving district-level coordinators, recruitment designed to cover demographic groups, questions and scenarios tailored to regions and domains, and a multi-stage transcription and quality-control pipeline.

The shift from Kathbath to IndicVoices is a shift from recording a language to recording how its speakers talk. It is also a large increase in operational complexity. Read speech needs a script and a microphone. Spontaneous speech from sixteen thousand people in 145 districts needs field teams, local trust, consent procedures, and transcribers who understand each dialect well enough to write down what was said, including the hesitations and the switches into other languages.

Transcribing spontaneous speech is an annotation task with every difficulty this series has discussed. Transcribers disagree on how to render dialect words, on where one language ends and another begins, and on how to spell words that have no standard written form. Guidelines have to settle these questions, and the answers shape what the model learns.

The self-supervision argument

A fair objection is that modern speech models need far less labelled data than this account implies.

Self-supervised methods learn representations of speech from unlabelled audio and then need only a modest amount of transcribed speech to reach good accuracy. An AI4Bharat study pretrained a model on about 17,000 hours of raw speech across 40 Indian languages and showed substantial gains. Meta’s Massively Multilingual Speech project extended recognition to more than a thousand languages, using recordings of religious texts as a source of aligned audio where nothing else existed.

These methods help. They reduce the quantity of transcription required. They do not remove the coverage requirement. Unlabelled audio still has to include the accents, conditions and speaking styles the model will meet. And a model trained largely on religious readings by a small number of speakers, as the MMS authors acknowledge, reflects that domain and those voices.

Evaluation is unaffected by any of this. To know the error rate for elderly women speaking Bhojpuri on a basic handset, somebody must have recorded and accurately transcribed elderly women speaking Bhojpuri on a basic handset.

Why this is a sovereignty question

Speech data has characteristics that make its governance more sensitive than text.

A voice recording is biometric. It can identify the speaker. Spontaneous speech about a person’s life contains personal information. India’s Digital Personal Data Protection Act of 2023 sets conditions for processing such data, centred on consent and purpose.

Speech data for Indian languages also has to be collected in India, from Indian speakers. It cannot be obtained elsewhere. That makes decisions about who collects it, under what terms, and who may use the result matters of public policy as well as commerce. The open release of datasets such as IndicVoices under permissive licences is one answer: publicly funded data as shared infrastructure. Government programmes including Bhashini and the IndiaAI Mission’s datasets platform follow the same reasoning.

Open datasets provide a foundation of general coverage. They do not cover an insurer’s claims vocabulary, a hospital’s consultations or a bank’s customer calls in six states. That domain-specific layer is for each organisation to build, and it carries the same obligations of consent and care.

Collecting speech data well

Several practices follow from the research.

Design the sample. Specify target counts by language, district, age band, gender and background before collection, and track them.

Prefer natural speech. Prompt speakers with questions and scenarios, not scripts, for a large share of the data.

Record in realistic conditions. Use the devices and environments in which the product will be used.

Obtain informed consent. Explain in the speaker’s language what the recording is for, who will use it and how to withdraw.

Pay contributors fairly. They are providing something that cannot be obtained otherwise.

Transcribe with native speakers of the dialect. Set and document conventions for dialect words, mixed-language segments, numerals and disfluencies.

Measure transcription agreement. Have a sample transcribed twice and compare. Disagreements reveal where conventions are unclear.

Evaluate by slice. Report error rates by language, region, gender, age and recording condition, since the average conceals the groups served worst.

What this changes