# A Deep Dive into the Disparity of Word Error Rates Across Thousands of NPTEL MOOC Videos

arXiv: https://arxiv.org/abs/2307.10587 · HTML: https://mlsanity.com/paper/2307.10587/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 75 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 17 of 17)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Adversarial Machine Learning -- Industry Perspectives](https://mlsanity.com/paper/2002.05646/) (2020)
- [Quantifying Bias in Automatic Speech Recognition](https://mlsanity.com/paper/2103.15122/) (2021)
- [Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding](https://mlsanity.com/paper/2104.02138/) (2021)
- [The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR](https://mlsanity.com/paper/2303.18110/) (2023)
- [Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance](https://mlsanity.com/paper/1908.08717/) (2019)
- [AccentDB: A Database of Non-Native English Accents to Assist Neural Speech Recognition](https://mlsanity.com/paper/2005.07973/) (2020)
- [Accented Speech Recognition: A Survey](https://mlsanity.com/paper/2104.10747/) (2021)
- [The History of Speech Recognition to the Year 2030](https://mlsanity.com/paper/2108.00084/) (2021)
- [Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learning](https://mlsanity.com/paper/2110.00678/) (2021)
- [Assessing ASR Model Quality on Disordered Speech using BERTScore](https://mlsanity.com/paper/2209.10591/) (2022)
- [Improving Automatic Speech Recognition for Non-Native English with Transfer Learning and Language Model Decoding](https://mlsanity.com/paper/2202.05209/) (2022)
- [Better Transcription of UK Supreme Court Hearings](https://mlsanity.com/paper/2211.17094/) (2022)
- [Decoding Demographic un-fairness from Indian Names](https://mlsanity.com/paper/2209.03089/) (2022)

## Built on by (most cited first, 5 of 5)

- [ASR-FAIRBENCH: Measuring and Benchmarking Equity Across Speech Recognition Systems](https://mlsanity.com/paper/2505.11572/) (2025)
- [DENOASR: Debiasing ASRs through Selective Denoising](https://mlsanity.com/paper/2410.16712/) (2024)
- [VideoConviction: A Multimodal Benchmark for Human Conviction and Stock Market Recommendations](https://mlsanity.com/paper/2507.08104/) (2025)
- [ASR Under the Stethoscope: Evaluating Biases in Clinical Speech Recognition across Indian Languages](https://mlsanity.com/paper/2512.10967/) (2025)
- [SamaVaani: Auditing and Debiasing Multilingual Clinical ASR for Indian Languages](https://mlsanity.com/paper/2606.26901/) (2026)

## Abstract (arXiv)

Automatic speech recognition (ASR) systems are designed to transcribe spoken language into written text and find utility in a variety of applications including voice assistants and transcription services. However, it has been observed that state-of-the-art ASR systems which deliver impressive benchmark results, struggle with speakers of certain regions or demographics due to variation in their speech properties. In this work, we describe the curation of a massive speech dataset of 8740 hours consisting of ~9.8K technical lectures in the English language along with their transcripts delivered by instructors representing various parts of Indian demography. The dataset is sourced from the very popular NPTEL MOOC platform. We use the curated dataset to measure the existing disparity in YouTube Automatic Captions and OpenAI Whisper model performance across the diverse demographic traits of speakers in India. While there exists disparity due to gender, native region, age and speech rate of speakers, disparity based on caste is non-existent. We also observe statistically significant disparity across the disciplines of the lectures. These results indicate the need of more inclusive and robust ASR systems and more representational datasets for disparity evaluation in them.