mlsanityData to 7 Oct 2026

What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations

arXiv 2409.02449 ↗First posted 4 Sep 2024cs.CL cs.AI cs.HCcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper explores the pitfalls in evaluating multilingual automatic speech recognition (ASR) models, with a particular focus on Indic language scripts. We investigate the text normalization routine employed by leading ASR models, including OpenAI Whisper, Meta's MMS, Seamless, and Assembly AI's Conformer, and their unintended consequences on performance metrics. Our research reveals that current text normalization practices, while aiming to standardize ASR outputs for fair comparison, by removing inconsistencies such as variations in spelling, punctuation, and special characters, are fundamentally flawed when applied to Indic scripts. Through empirical analysis using text similarity scores and in-depth linguistic examination, we demonstrate that these flaws lead to artificially improved performance metrics for Indic languages. We conclude by proposing a shift towards developing text normalization routines that leverage native linguistic expertise, ensuring more robust and accurate evaluations of multilingual ASR models.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 7 of its 21 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

7 of 21 references.

References · 7

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  3. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  4. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  5. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  6. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  7. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 8.

12 papers.

  1. 2024Advocating Character Error Rate for Multilingual ASR Evaluationcited by 22
  2. 2024Multistage Fine-tuning Strategies for Automatic Speech Recognition in Low-resource Languagescited by 6
  3. 2026Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languagescited by 5
  4. 2025Parallel Corpora for Machine Translation in Low-resource Indic Languages: A Comprehensive Reviewcited by 4
  5. 2025Synthetic Voice Data for Automatic Speech Recognition in African Languagescited by 3
  6. 2026Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluationcited by 2
  7. 2026Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMscited by 1
  8. 2026BlasBench: An Open Benchmark for Irish Speech Recognitioncited by 0
  9. 2026Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in Indiacited by 0
  10. 2026SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASRcited by 0
  11. 2026SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluationcited by 0
  12. 2026The Hidden Cost of Digits: Number Normalization and WER in ASR Systemscited by 0