# Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Models

arXiv: https://arxiv.org/abs/2211.05103 · HTML: https://mlsanity.com/paper/2211.05103/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 9 Nov 2022.

## Builds on (most cited first, 14 of 14)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [VoxLingua107: a Dataset for Spoken Language Recognition](https://mlsanity.com/paper/2011.12998/) (2020)
- [Exploring wav2vec 2.0 on speaker verification and language identification](https://mlsanity.com/paper/2012.06185/) (2020)
- [Towards Building ASR Systems for the Next Billion Users](https://mlsanity.com/paper/2111.03945/) (2021)
- [Improved Language Identification Through Cross-Lingual Self-Supervised Learning](https://mlsanity.com/paper/2107.04082/) (2021)
- [BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognition](https://mlsanity.com/paper/1907.00457/) (2019)
- [Improving Language Identification of Accented Speech](https://mlsanity.com/paper/2203.16972/) (2022)

## Built on by (most cited first, 6 of 6)

- [Findings of the VarDial Evaluation Campaign 2023](https://mlsanity.com/paper/2305.20080/) (2023)
- [Speech Analysis of Language Varieties in Italy](https://mlsanity.com/paper/2406.15862/) (2024)
- [CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing](https://mlsanity.com/paper/2412.04425/) (2024)
- [Efficiency-oriented approaches for self-supervised speech representation learning](https://mlsanity.com/paper/2312.11142/) (2023)
- [Leveraging Zipformer Model for Effective Language Identification in Code-Switched Child-Directed Speech](https://mlsanity.com/paper/2508.09430/) (2025)
- [Anuvani-LID: Robust and Calibrated Spoken Language Identification for Indian Languages](https://mlsanity.com/paper/2610.20263/) (2026)

## Abstract (arXiv)

In this paper, we extend previous self-supervised approaches for language identification by experimenting with Conformer based architecture in a multilingual pre-training paradigm. We find that pre-trained speech models optimally encode language discriminatory information in lower layers. Further, we demonstrate that the embeddings obtained from these layers are significantly robust to classify unseen languages and different acoustic environments without additional training. After fine-tuning a pre-trained Conformer model on the VoxLingua107 dataset, we achieve results similar to current state-of-the-art systems for language identification. More, our model accomplishes this with 5x less parameters. We open-source the model through the NVIDIA NeMo toolkit.