# Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India

arXiv: https://arxiv.org/abs/2604.19151 · HTML: https://mlsanity.com/paper/2604.19151/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Builds on (most cited first, 21 of 21)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [Rethinking Evaluation in ASR: Are Our Models Robust Enough?](https://mlsanity.com/paper/2010.11745/) (2020)
- [Multilingual and code-switching ASR challenges for low resource Indian languages](https://mlsanity.com/paper/2104.00235/) (2021)
- [IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages](https://mlsanity.com/paper/2403.01926/) (2024)
- [IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages](https://mlsanity.com/paper/2208.11761/) (2022)
- [Advocating Character Error Rate for Multilingual ASR Evaluation](https://mlsanity.com/paper/2410.07400/) (2024)
- [Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR](https://mlsanity.com/paper/2305.15386/) (2023)
- [Svarah: Evaluating English ASR Systems on Indian Accents](https://mlsanity.com/paper/2305.15760/) (2023)
- [L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT](https://mlsanity.com/paper/2304.11434/) (2023)
- [What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations](https://mlsanity.com/paper/2409.02449/) (2024)
- [Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency](https://mlsanity.com/paper/2306.04530/) (2023)
- [LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems](https://mlsanity.com/paper/2408.11440/) (2024)
- [WERd: Using Social Text Spelling Variants for Evaluating Dialectal Speech Recognition](https://mlsanity.com/paper/1709.07484/) (2017)
- [Google Crowdsourced Speech Corpora and Related Open-Source Resources for Low-Resource Languages and Dialects: An Overview](https://mlsanity.com/paper/2010.06778/) (2020)
- [Style-agnostic evaluation of ASR using multiple reference transcripts](https://mlsanity.com/paper/2412.07937/) (2024)
- [Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages](https://mlsanity.com/paper/2603.00941/) (2026)
- [Unsupervised Language agnostic WER Standardization](https://mlsanity.com/paper/2303.05046/) (2023)

## Abstract (arXiv)

Existing Indic ASR benchmarks often use scripted, clean speech and leaderboard driven evaluation that encourages dataset specific overfitting. In addition, strict single reference WER penalizes natural spelling variation in Indian languages, including non standardized spellings of code-mixed English origin words. To address these limitations, we introduce Voice of India, a closed source benchmark built from unscripted telephonic conversations covering 15 major Indian languages across 139 regional clusters. The dataset contains 306230 utterances, totaling 536 hours of speech from 36691 speakers with transcripts accounting for spelling variations. We also analyze performance geographically at the district level, revealing disparities. Finally, we provide detailed analysis across factors such as audio quality, speaking rate, gender, and device type, highlighting where current ASR systems struggle and offering insights for improving real world Indic ASR systems.