# Towards Building ASR Systems for the Next Billion Users

arXiv: https://arxiv.org/abs/2111.03945 · HTML: https://mlsanity.com/paper/2111.03945/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 44 references that are in the map.
- First posted on arXiv 6 Nov 2021.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages](https://mlsanity.com/paper/2104.05596/) (2021)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Iterative Pseudo-Labeling for Speech Recognition](https://mlsanity.com/paper/2005.09267/) (2020)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)
- [Effectiveness of self-supervised pre-training for speech recognition](https://mlsanity.com/paper/1911.03912/) (2019)
- [Multilingual and code-switching ASR challenges for low resource Indian languages](https://mlsanity.com/paper/2104.00235/) (2021)
- [Transformers with convolutional context for ASR](https://mlsanity.com/paper/1904.11660/) (2019)
- [wav2letter++: The Fastest Open-source Speech Recognition System](https://mlsanity.com/paper/1812.07625/) (2018)
- [CLSRIL-23: Cross Lingual Speech Representations for Indic Languages](https://mlsanity.com/paper/2107.07402/) (2021)
- [The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERT](https://mlsanity.com/paper/2101.09115/) (2021)
- [Letter-Based Speech Recognition with Gated ConvNets](https://mlsanity.com/paper/1712.09444/) (2017)
- [Utilizing Language Relatedness to improve Machine Translation: A Case Study on Languages of the Indian Subcontinent](https://mlsanity.com/paper/2003.08925/) (2020)

## Built on by (most cited first, 30 of 47)

- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages](https://mlsanity.com/paper/2208.11761/) (2022)
- [AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASR](https://mlsanity.com/paper/2310.00274/) (2023)
- [Towards Building Text-To-Speech Systems for the Next Billion Users](https://mlsanity.com/paper/2211.09536/) (2022)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [A Survey of Multilingual Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.12576/) (2022)
- [Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR](https://mlsanity.com/paper/2305.15386/) (2023)
- [Effectiveness of Mining Audio and Text Pairs from Public Data for Improving ASR Systems for Low-Resource Languages](https://mlsanity.com/paper/2208.12666/) (2022)
- [BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric](https://mlsanity.com/paper/2212.08486/) (2022)
- [Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokens](https://mlsanity.com/paper/2402.02302/) (2024)
- [AfriNames: Most ASR models "butcher" African Names](https://mlsanity.com/paper/2306.00253/) (2023)
- [Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Models](https://mlsanity.com/paper/2211.05103/) (2022)
- [Empowering Low-Resource Language ASR via Large-Scale Pseudo Labeling](https://mlsanity.com/paper/2408.14026/) (2024)
- [Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages](https://mlsanity.com/paper/2603.00941/) (2026)
- [Label Aware Speech Representation Learning For Language Identification](https://mlsanity.com/paper/2306.04374/) (2023)
- [CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving](https://mlsanity.com/paper/2406.10993/) (2024)
- [Textless Speech-to-Speech Translation With Limited Parallel Data](https://mlsanity.com/paper/2305.15405/) (2023)
- [BanglaTalk: Towards Real-Time Speech Assistance for Bengali Regional Dialects](https://mlsanity.com/paper/2510.06188/) (2025)
- [Analyzing the factors affecting usefulness of Self-Supervised Pre-trained Representations for Speech Recognition](https://mlsanity.com/paper/2203.16973/) (2022)
- [An Automatic Speech Recognition System for Bengali Language based on Wav2Vec2 and Transfer Learning](https://mlsanity.com/paper/2209.08119/) (2022)
- [Vedavani: A Benchmark Corpus for ASR on Vedic Sanskrit Poetry](https://mlsanity.com/paper/2506.00145/) (2025)
- [Beyond WER: Probing Whisper's Sub-token Decoder Across Diverse Language Resource Levels](https://mlsanity.com/paper/2509.25516/) (2025)
- [Benchmarking Automatic Speech Recognition for Indian Languages in Agricultural Contexts](https://mlsanity.com/paper/2602.03868/) (2026)
- [PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech](https://mlsanity.com/paper/2604.25476/) (2026)
- [Seeds Before Objectives: Rethinking Evaluation for Low-Resource Garhwali ASR](https://mlsanity.com/paper/2608.10670/) (2026)
- [WavFT: Acoustic model finetuning with labelled and unlabelled data](https://mlsanity.com/paper/2204.00348/) (2022)
- [MASR: Multi-label Aware Speech Representation](https://mlsanity.com/paper/2307.10982/) (2023)
- ["We care": Improving Code Mixed Speech Emotion Recognition in Customer-Care Conversations](https://mlsanity.com/paper/2308.03150/) (2023)
- [Multimodal Modeling For Spoken Language Identification](https://mlsanity.com/paper/2309.10567/) (2023)
- [Temporally Aligning Long Audio Interviews with Questions: A Case Study in Multimodal Data Integration](https://mlsanity.com/paper/2310.06702/) (2023)

## Abstract (arXiv)

Recent methods in speech and language technology pretrain very LARGE models which are fine-tuned for specific tasks. However, the benefits of such LARGE models are often limited to a few resource rich languages of the world. In this work, we make multiple contributions towards building ASR systems for low resource languages from the Indian subcontinent. First, we curate 17,000 hours of raw speech data for 40 Indian languages from a wide variety of domains including education, news, technology, and finance. Second, using this raw speech data we pretrain several variants of wav2vec style models for 40 Indian languages. Third, we analyze the pretrained models to find key features: codebook vectors of similar sounding phonemes are shared across languages, representations across layers are discriminative of the language family, and attention heads often pay attention within small local windows. Fourth, we fine-tune this model for downstream ASR for 9 languages and obtain state-of-the-art results on 3 public datasets, including on very low-resource languages such as Sinhala and Nepali. Our work establishes that multilingual pretraining is an effective strategy for building ASR systems for the linguistically diverse speakers of the Indian subcontinent. Our code, data and models are available publicly at https://indicnlp.ai4bharat.org/indicwav2vec/ and we hope they will help advance research in ASR for Indic languages.