# CLSRIL-23: Cross Lingual Speech Representations for Indic Languages

arXiv: https://arxiv.org/abs/2107.07402 · HTML: https://mlsanity.com/paper/2107.07402/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 15 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech](https://mlsanity.com/paper/1803.08976/) (2018)
- [Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech](https://mlsanity.com/paper/1911.09602/) (2019)
- [Improving Transformer-based Speech Recognition Using Unsupervised Pre-training](https://mlsanity.com/paper/1910.09932/) (2019)
- [VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019](https://mlsanity.com/paper/1905.11449/) (2019)

## Built on by (most cited first, 22 of 22)

- [Towards Building ASR Systems for the Next Billion Users](https://mlsanity.com/paper/2111.03945/) (2021)
- [A Survey of Multilingual Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.12576/) (2022)
- [ADIMA: Abuse Detection In Multilingual Audio](https://mlsanity.com/paper/2202.07991/) (2022)
- [3MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videos](https://mlsanity.com/paper/2203.14456/) (2022)
- [Vakyansh: ASR Toolkit for Low Resource Indic languages](https://mlsanity.com/paper/2203.16512/) (2022)
- [Enhancing Whisper's Accuracy and Speed for Indian Languages through Prompt-Tuning and Tokenization](https://mlsanity.com/paper/2412.19785/) (2024)
- [Code Switched and Code Mixed Speech Recognition for Indic languages](https://mlsanity.com/paper/2203.16578/) (2022)
- [Analyzing the factors affecting usefulness of Self-Supervised Pre-trained Representations for Speech Recognition](https://mlsanity.com/paper/2203.16973/) (2022)
- [VAKTA-SETU: A Speech-to-Speech Machine Translation Service in Select Indic Languages](https://mlsanity.com/paper/2305.12518/) (2023)
- [Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition](https://mlsanity.com/paper/2312.12783/) (2023)
- [Improving Speech Recognition for Indic Languages using Language Model](https://mlsanity.com/paper/2203.16595/) (2022)
- [Implicit Self-supervised Language Representation for Spoken Language Diarization](https://mlsanity.com/paper/2308.10470/) (2023)
- [Dialect Adaptation and Data Augmentation for Low-Resource ASR: TalTech Systems for the MADASR 2023 Challenge](https://mlsanity.com/paper/2310.17448/) (2023)
- [FusDom: Combining In-Domain and Out-of-Domain Knowledge for Continuous Self-Supervised Learning](https://mlsanity.com/paper/2312.13026/) (2023)
- [Low-Resource Transliteration for Roman-Urdu and Urdu Using Transformer-Based Models](https://mlsanity.com/paper/2503.21530/) (2025)
- [On the Cross-lingual Transferability of Pre-trained wav2vec2-based Models](https://mlsanity.com/paper/2511.21704/) (2025)
- [Is Word Error Rate a good evaluation metric for Speech Recognition in Indic Languages?](https://mlsanity.com/paper/2203.16601/) (2022)
- [Effectiveness of text to speech pseudo labels for forced alignment and cross lingual pretrained models for low resource speech recognition](https://mlsanity.com/paper/2203.16823/) (2022)
- [Multilingual and Multimodal Abuse Detection](https://mlsanity.com/paper/2204.02263/) (2022)
- [Implicit spoken language diarization](https://mlsanity.com/paper/2306.12913/) (2023)
- [Temporally Aligning Long Audio Interviews with Questions: A Case Study in Multimodal Data Integration](https://mlsanity.com/paper/2310.06702/) (2023)
- [Towards Cross-Lingual Audio Abuse Detection in Low-Resource Settings with Few-Shot Learning](https://mlsanity.com/paper/2412.01408/) (2024)

## Abstract (arXiv)

We present a CLSRIL-23, a self supervised learning based audio pre-trained model which learns cross lingual speech representations from raw audio across 23 Indic languages. It is built on top of wav2vec 2.0 which is solved by training a contrastive task over masked latent speech representations and jointly learns the quantization of latents shared across all languages. We compare the language wise loss during pretraining to compare effects of monolingual and multilingual pretraining. Performance on some downstream fine-tuning tasks for speech recognition is also compared and our experiments show that multilingual pretraining outperforms monolingual training, in terms of learning speech representations which encodes phonetic similarity of languages and also in terms of performance on down stream tasks. A decrease of 5% is observed in WER and 9.5% in CER when a multilingual pretrained model is used for finetuning in Hindi. All the code models are also open sourced. CLSRIL-23 is a model trained on 23 languages and almost 10,000 hours of audio data to facilitate research in speech recognition for Indic languages. We hope that new state of the art systems will be created using the self supervised approach, especially for low resources Indic languages.