# Advocating Character Error Rate for Multilingual ASR Evaluation

arXiv: https://arxiv.org/abs/2410.07400 · HTML: https://mlsanity.com/paper/2410.07400/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 37 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Builds on (most cited first, 8 of 8)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding](https://mlsanity.com/paper/2104.02138/) (2021)
- [Automatic Speech Recognition in Sanskrit: A New Speech Corpus and Modelling Insights](https://mlsanity.com/paper/2106.05852/) (2021)
- [What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations](https://mlsanity.com/paper/2409.02449/) (2024)

## Built on by (most cited first, 24 of 24)

- [SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Dataset](https://mlsanity.com/paper/2506.00087/) (2025)
- [Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understanding](https://mlsanity.com/paper/2602.01785/) (2026)
- [Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages](https://mlsanity.com/paper/2603.00941/) (2026)
- [Evaluation of Automatic Speech Recognition Using Generative Large Language Models](https://mlsanity.com/paper/2604.21928/) (2026)
- [The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?](https://mlsanity.com/paper/2606.07435/) (2026)
- [Linguistically Informed Evaluation of Multilingual ASR for African Languages](https://mlsanity.com/paper/2602.04716/) (2026)
- [Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation](https://mlsanity.com/paper/2604.04598/) (2026)
- [Romanized to Native Malayalam Script Transliteration Using an Encoder-Decoder Framework](https://mlsanity.com/paper/2412.09957/) (2024)
- [Beyond Word Error Rate: Auditing the Diversity Tax in Speech Recognition through Dataset Cartography](https://mlsanity.com/paper/2603.05267/) (2026)
- [WAXAL-NET: Finetuned Edge ASR Across 19 African Languages](https://mlsanity.com/paper/2606.02375/) (2026)
- [grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP](https://mlsanity.com/paper/2607.22456/) (2026)
- [Automatic Correction of Writing Anomalies in Hausa Texts](https://mlsanity.com/paper/2506.03820/) (2025)
- [PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMs](https://mlsanity.com/paper/2509.19745/) (2025)
- [Seeing Justice Clearly: Handwritten Legal Document Translation with OCR and Vision-Language Models](https://mlsanity.com/paper/2512.18004/) (2025)
- [Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India](https://mlsanity.com/paper/2604.19151/) (2026)
- [RAS: a Reliability Oriented Metric for Automatic Speech Recognition](https://mlsanity.com/paper/2604.24278/) (2026)
- [SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR](https://mlsanity.com/paper/2605.20712/) (2026)
- [SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation](https://mlsanity.com/paper/2606.02548/) (2026)
- [Building and Evaluating a Synthetic Bengali Speech Resource for Telecom Customer Care](https://mlsanity.com/paper/2608.20346/) (2026)
- [Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study](https://mlsanity.com/paper/2608.25574/) (2026)
- [Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER](https://mlsanity.com/paper/2609.21663/) (2026)
- [Benchmarking and Domain Adaptation of Automatic Speech Recognition (ASR) for Adolescent Health Communication in Ghanaian Languages](https://mlsanity.com/paper/2609.29798/) (2026)
- [Personalized Automatic Speech Recognition for a Dysarthric and Tracheostomic Speaker using Artificial Conversations](https://mlsanity.com/paper/2610.03017/) (2026)
- [Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR](https://mlsanity.com/paper/2610.20461/) (2026)

## Abstract (arXiv)

Automatic speech recognition (ASR) systems have traditionally been evaluated using English datasets, with the word error rate (WER) serving as the predominant metric. WER's simplicity and ease of interpretation have contributed to its widespread adoption, particularly for English. However, as ASR systems expand to multilingual contexts, WER fails in various ways, particularly with morphologically complex languages or those without clear word boundaries. Our work documents the limitations of WER as an evaluation metric and advocates for the character error rate (CER) as the primary metric in multilingual ASR evaluation. We show that CER avoids many of the challenges WER faces and exhibits greater consistency across writing systems. We support our proposition by conducting human evaluations of ASR transcriptions in three languages: Malayalam, English, and Arabic, which exhibit distinct morphological characteristics. We show that CER correlates more closely with human judgments than WER, even for English. To facilitate further research, we release our human evaluation dataset for future benchmarking of ASR metrics. Our findings suggest that CER should be prioritized, or at least supplemented, in multilingual ASR evaluations to account for the varying linguistic characteristics of different languages.