# LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems

arXiv: https://arxiv.org/abs/2408.11440 · HTML: https://mlsanity.com/paper/2408.11440/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Multilingual and code-switching ASR challenges for low resource Indian languages](https://mlsanity.com/paper/2104.00235/) (2021)
- [IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages](https://mlsanity.com/paper/2403.01926/) (2024)
- [IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages](https://mlsanity.com/paper/2208.11761/) (2022)
- [Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR](https://mlsanity.com/paper/2305.15386/) (2023)
- [Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users](https://mlsanity.com/paper/2205.03018/) (2022)
- [Svarah: Evaluating English ASR Systems on Indian Accents](https://mlsanity.com/paper/2305.15760/) (2023)
- [A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategies](https://mlsanity.com/paper/2201.05420/) (2022)
- [SPRING-INX: A Multilingual Indian Language Speech Corpus by SPRING Lab, IIT Madras](https://mlsanity.com/paper/2310.14654/) (2023)

## Built on by (most cited first, 10 of 10)

- [VoiceAgentBench: Are Voice Assistants ready for agentic tasks?](https://mlsanity.com/paper/2510.07978/) (2025)
- [NIRANTAR: Continual Learning with New Languages and Domains on Real-world Speech Data](https://mlsanity.com/paper/2507.00534/) (2025)
- [BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources](https://mlsanity.com/paper/2604.18423/) (2026)
- [INDIC DIALECT: A Multi Task Benchmark to Evaluate and Translate in Indian Language Dialects](https://mlsanity.com/paper/2601.10388/) (2026)
- [Task-Lens: Cross-Task Utility Based Speech Dataset Profiling for Low-Resource Indian Languages](https://mlsanity.com/paper/2602.23388/) (2026)
- [Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India](https://mlsanity.com/paper/2604.19151/) (2026)
- [Convex Low-resource Accent-Robust Language Detection in Speech Recognition](https://mlsanity.com/paper/2605.23235/) (2026)
- [Position: AI Leaderboards Are Underserving the Global South: A Case Study from India](https://mlsanity.com/paper/2608.18117/) (2026)
- [Vimarsha: Faithful ASR Evaluation for Indian Languages with Demographic Diversity, In-the-Wild Audio and Spelling Variations](https://mlsanity.com/paper/2609.24199/) (2026)
- [Anuvani-LID: Robust and Calibrated Spoken Language Identification for Indian Languages](https://mlsanity.com/paper/2610.20263/) (2026)

## Abstract (arXiv)

Hindi, one of the most spoken language of India, exhibits a diverse array of accents due to its usage among individuals from diverse linguistic origins. To enable a robust evaluation of Hindi ASR systems on multiple accents, we create a benchmark, LAHAJA, which contains read and extempore speech on a diverse set of topics and use cases, with a total of 12.5 hours of Hindi audio, sourced from 132 speakers spanning 83 districts of India. We evaluate existing open-source and commercial models on LAHAJA and find their performance to be poor. We then train models using different datasets and find that our model trained on multilingual data with good speaker diversity outperforms existing models by a significant margin. We also present a fine-grained analysis which shows that the performance declines for speakers from North-East and South India, especially with content heavy in named entities and specialized terminology.