mlsanityData to 7 Oct 2026

Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages

arXiv 2603.00941 ↗First posted 1 Mar 2026cs.CL cs.SDcited by 5

Not on the map. Why?

Abstract (arXiv)

Evaluating ASR systems for Indian languages is challenging due to spelling variations, suffix splitting flexibility, and non-standard spellings in code-mixed words. Traditional Word Error Rate (WER) often presents a bleaker picture of system performance than what human users perceive. Better aligning evaluation with real-world performance requires capturing permissible orthographic variations, which is extremely challenging for under-resourced Indian languages. Leveraging recent advances in LLMs, we propose a framework for creating benchmarks that capture permissible variations. Through extensive experiments, we demonstrate that OIWER, by accounting for orthographic variations, reduces pessimistic error rates (an average improvement of 6.3 points), narrows inflated model gaps (e.g., Gemini-Canary performance difference drops from 18.1 to 11.5 points), and aligns more closely with human perception than prior methods like WER-SN by 4.9 points.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 14 of its 23 references that are in the map. First posted on arXiv 1 Mar 2026.

Builds on

14 of 23 references.

References · 14

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  3. 2021Towards Building ASR Systems for the Next Billion Userscited by 45
  4. 2024IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languagescited by 39
  5. 2022IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languagescited by 38
  6. 2024Advocating Character Error Rate for Multilingual ASR Evaluationcited by 22
  7. 2023Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASRcited by 21
  8. 2024What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluationscited by 12
  9. 2023Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistencycited by 10
  10. 2025Training and Inference Efficiency of Encoder-Decoder Speech Modelscited by 8
  11. 2017WERd: Using Social Text Spelling Variants for Evaluating Dialectal Speech Recognitioncited by 6
  12. 2024Style-agnostic evaluation of ASR using multiple reference transcriptscited by 4
2 more
  1. 2023Unsupervised Language agnostic WER Standardizationcited by 3
  2. 2024Quantification of stylistic differences in human- and ASR-produced transcripts of African American Englishcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in Indiacited by 0
  2. 2026SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASRcited by 0
  3. 2026Evaluation of forced alignment of code-mixed speech: the case of Hindi-Englishcited by 0
  4. 2026Vimarsha: Faithful ASR Evaluation for Indian Languages with Demographic Diversity, In-the-Wild Audio and Spelling Variationscited by 0
  5. 2026A Training Criterion with Token-Level Tolerance to Transcription Ambiguity for Automatic Speech Recognitioncited by 0