# WER We Stand: Benchmarking Urdu ASR Models

arXiv: https://arxiv.org/abs/2409.11252 · HTML: https://mlsanity.com/paper/2409.11252/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 17 Sep 2024.

## Builds on (most cited first, 9 of 9)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Multilingual Speech Recognition With A Single End-To-End Model](https://mlsanity.com/paper/1711.01694/) (2017)
- [IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages](https://mlsanity.com/paper/2208.11761/) (2022)
- [Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency](https://mlsanity.com/paper/2306.04530/) (2023)
- [Code-Switched Urdu ASR for Noisy Telephonic Environment using Data Centric Approach with Hybrid HMM and CNN-TDNN](https://mlsanity.com/paper/2307.12759/) (2023)

## Built on by (most cited first, 8 of 8)

- [CAM~OES: A Comprehensive Automatic Speech Recognition Benchmark for European Portuguese](https://mlsanity.com/paper/2508.19721/) (2025)
- [Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation](https://mlsanity.com/paper/2604.04598/) (2026)
- [Efficient ASR for Low-Resource Languages: Leveraging Cross-Lingual Unlabeled Data](https://mlsanity.com/paper/2512.07277/) (2025)
- [Assessing the Feasibility of Lightweight Whisper Models for Low-Resource Urdu Transcription](https://mlsanity.com/paper/2508.09865/) (2025)
- [SD-MVSum: Script-Driven Multimodal Video Summarization Method and Datasets](https://mlsanity.com/paper/2510.05652/) (2025)
- [SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation](https://mlsanity.com/paper/2606.02548/) (2026)
- [Neyshekar: An Open Persian Read-Speech Corpus for Automatic Speech Recognition](https://mlsanity.com/paper/2609.14542/) (2026)
- [Multi-Task Learning by using Contextualized Word Representations for Syntactic Parsing of a Morphologically Rich Language](https://mlsanity.com/paper/2609.29855/) (2026)

## Abstract (arXiv)

This paper presents a comprehensive evaluation of Urdu Automatic Speech Recognition (ASR) models. We analyze the performance of three ASR model families: Whisper, MMS, and Seamless-M4T using Word Error Rate (WER), along with a detailed examination of the most frequent wrong words and error types including insertions, deletions, and substitutions. Our analysis is conducted using two types of datasets, read speech and conversational speech. Notably, we present the first conversational speech dataset designed for benchmarking Urdu ASR models. We find that seamless-large outperforms other ASR models on the read speech dataset, while whisper-large performs best on the conversational speech dataset. Furthermore, this evaluation highlights the complexities of assessing ASR models for low-resource languages like Urdu using quantitative metrics alone and emphasizes the need for a robust Urdu text normalization system. Our findings contribute valuable insights for developing robust ASR systems for low-resource languages like Urdu.