mlsanityData to 7 Oct 2026

Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India

arXiv 2604.19151 ↗First posted 21 Apr 2026cs.CL cs.SD eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Existing Indic ASR benchmarks often use scripted, clean speech and leaderboard driven evaluation that encourages dataset specific overfitting. In addition, strict single reference WER penalizes natural spelling variation in Indian languages, including non standardized spellings of code-mixed English origin words. To address these limitations, we introduce Voice of India, a closed source benchmark built from unscripted telephonic conversations covering 15 major Indian languages across 139 regional clusters. The dataset contains 306230 utterances, totaling 536 hours of speech from 36691 speakers with transcripts accounting for spelling variations. We also analyze performance geographically at the district level, revealing disparities. Finally, we provide detailed analysis across factors such as audio quality, speaking rate, gender, and device type, highlighting where current ASR systems struggle and offering insights for improving real world Indic ASR systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 32 references that are in the map. First posted on arXiv 21 Apr 2026.

Builds on

21 of 32 references.

References · 21

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  3. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  4. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  5. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 235
  6. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 72
  7. 2021Multilingual and code-switching ASR challenges for low resource Indian languagescited by 50
  8. 2024IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languagescited by 39
  9. 2022IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languagescited by 38
  10. 2024Advocating Character Error Rate for Multilingual ASR Evaluationcited by 22
  11. 2023Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASRcited by 21
  12. 2023L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERTcited by 15
9 more
  1. 2023Svarah: Evaluating English ASR Systems on Indian Accentscited by 15
  2. 2024What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluationscited by 12
  3. 2023Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistencycited by 10
  4. 2024LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systemscited by 8
  5. 2017WERd: Using Social Text Spelling Variants for Evaluating Dialectal Speech Recognitioncited by 6
  6. 2020Google Crowdsourced Speech Corpora and Related Open-Source Resources for Low-Resource Languages and Dialects: An Overviewcited by 5
  7. 2026Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languagescited by 5
  8. 2024Style-agnostic evaluation of ASR using multiple reference transcriptscited by 4
  9. 2023Unsupervised Language agnostic WER Standardizationcited by 3

Built on by

Not cited yet.