# IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages

arXiv: https://arxiv.org/abs/2208.11761 · HTML: https://mlsanity.com/paper/2208.11761/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 24 Aug 2022.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [How Useful is Self-Supervised Pretraining for Visual Tasks?](https://mlsanity.com/paper/2003.14323/) (2020)
- [SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities](https://mlsanity.com/paper/2203.06849/) (2022)
- [Multilingual and code-switching ASR challenges for low resource Indian languages](https://mlsanity.com/paper/2104.00235/) (2021)
- [Towards Building ASR Systems for the Next Billion Users](https://mlsanity.com/paper/2111.03945/) (2021)
- [Automatic Speech Recognition in Sanskrit: A New Speech Corpus and Modelling Insights](https://mlsanity.com/paper/2106.05852/) (2021)
- [Subword Dictionary Learning and Segmentation Techniques for Automatic Speech Recognition in Tamil and Kannada](https://mlsanity.com/paper/2207.13331/) (2022)

## Built on by (most cited first, 30 of 40)

- [Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities](https://mlsanity.com/paper/2503.04721/) (2025)
- [ML-SUPERB: Multilingual Speech Universal PERformance Benchmark](https://mlsanity.com/paper/2305.10615/) (2023)
- [ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets](https://mlsanity.com/paper/2406.08641/) (2024)
- [Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR](https://mlsanity.com/paper/2305.15386/) (2023)
- [LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speech](https://mlsanity.com/paper/2309.05472/) (2023)
- [CL-MASR: A Continual Learning Benchmark for Multilingual ASR](https://mlsanity.com/paper/2310.16931/) (2023)
- [Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology](https://mlsanity.com/paper/2405.06346/) (2024)
- [Effectiveness of Mining Audio and Text Pairs from Public Data for Improving ASR Systems for Low-Resource Languages](https://mlsanity.com/paper/2208.12666/) (2022)
- [Svarah: Evaluating English ASR Systems on Indian Accents](https://mlsanity.com/paper/2305.15760/) (2023)
- [IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTS](https://mlsanity.com/paper/2409.05356/) (2024)
- [Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokens](https://mlsanity.com/paper/2402.02302/) (2024)
- [Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond](https://mlsanity.com/paper/2310.05513/) (2023)
- [LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems](https://mlsanity.com/paper/2408.11440/) (2024)
- [WER We Stand: Benchmarking Urdu ASR Models](https://mlsanity.com/paper/2409.11252/) (2024)
- [SVeritas: Benchmark for Robust Speaker Verification under Diverse Conditions](https://mlsanity.com/paper/2509.17091/) (2025)
- [Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages](https://mlsanity.com/paper/2603.00941/) (2026)
- [CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving](https://mlsanity.com/paper/2406.10993/) (2024)
- [ASR Under the Stethoscope: Evaluating Biases in Clinical Speech Recognition across Indian Languages](https://mlsanity.com/paper/2512.10967/) (2025)
- [Vedavani: A Benchmark Corpus for ASR on Vedic Sanskrit Poetry](https://mlsanity.com/paper/2506.00145/) (2025)
- [NIRANTAR: Continual Learning with New Languages and Domains on Real-world Speech Data](https://mlsanity.com/paper/2507.00534/) (2025)
- [TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End ASR](https://mlsanity.com/paper/2401.03251/) (2024)
- [Direct Punjabi to English speech translation using discrete units](https://mlsanity.com/paper/2402.15967/) (2024)
- [Dynamic Multi-Expert Projectors with Stabilized Routing for Multilingual Speech Recognition](https://mlsanity.com/paper/2601.19451/) (2026)
- [HuBERTopic: Enhancing Semantic Representation of HuBERT through Self-supervision Utilizing Topic Model](https://mlsanity.com/paper/2310.03975/) (2023)
- [The Faetar Benchmark: Speech Recognition in a Very Under-Resourced Language](https://mlsanity.com/paper/2409.08103/) (2024)
- [Navigating the Reality Gap: On-Device Continual Adaptation of ASR for Clinical Telephony](https://mlsanity.com/paper/2512.16401/) (2025)
- [Windowed SummaryMixing: An Efficient Fine-Tuning of Self-Supervised Learning Models for Low-resource Speech Recognition](https://mlsanity.com/paper/2602.09043/) (2026)
- [Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India](https://mlsanity.com/paper/2604.19151/) (2026)
- [Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition](https://mlsanity.com/paper/2605.13087/) (2026)
- [SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR](https://mlsanity.com/paper/2605.20712/) (2026)

## Abstract (arXiv)

A cornerstone in AI research has been the creation and adoption of standardized training and test datasets to earmark the progress of state-of-the-art models. A particularly successful example is the GLUE dataset for training and evaluating Natural Language Understanding (NLU) models for English. The large body of research around self-supervised BERT-based language models revolved around performance improvements on NLU tasks in GLUE. To evaluate language models in other languages, several language-specific GLUE datasets were created. The area of speech language understanding (SLU) has followed a similar trajectory. The success of large self-supervised models such as wav2vec2 enable creation of speech models with relatively easy to access unlabelled data. These models can then be evaluated on SLU tasks, such as the SUPERB benchmark. In this work, we extend this to Indic languages by releasing the IndicSUPERB benchmark. Specifically, we make the following three contributions. (i) We collect Kathbath containing 1,684 hours of labelled speech data across 12 Indian languages from 1,218 contributors located in 203 districts in India. (ii) Using Kathbath, we create benchmarks across 6 speech tasks: Automatic Speech Recognition, Speaker Verification, Speaker Identification (mono/multi), Language Identification, Query By Example, and Keyword Spotting for 12 languages. (iii) On the released benchmarks, we train and evaluate different self-supervised models alongside a commonly used baseline FBANK. We show that language-specific fine-tuned models are more accurate than baseline on most of the tasks, including a large gap of 76% for the Language Identification task. However, for speaker identification, self-supervised models trained on large datasets demonstrate an advantage. We hope IndicSUPERB contributes to the progress of developing speech language understanding models for Indian languages.