mlsanityData to 7 Oct 2026

Improved Language Identification Through Cross-Lingual Self-Supervised Learning

arXiv 2107.04082 ↗First posted 8 Jul 2021cs.CL cs.SD eess.AScited by 20

Not on the map. Why?

Abstract (arXiv)

Language identification greatly impacts the success of downstream tasks such as automatic speech recognition. Recently, self-supervised speech representations learned by wav2vec 2.0 have been shown to be very effective for a range of speech tasks. We extend previous self-supervised work on language identification by experimenting with pre-trained models which were learned on real-world unconstrained speech in multiple languages and not just on English. We show that models pre-trained on many languages perform better and enable language identification systems that require very little labeled data to perform well. Results on a 26 languages setup show that with only 10 minutes of labeled data per language, a cross-lingually pre-trained model can achieve over 89.2% accuracy.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 8 Jul 2021.

Builds on

25 of 36 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2016Layer Normalizationcited by 5,579
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2020On Layer Normalization in the Transformer Architecturecited by 906
  10. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  11. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  12. 2017A Structured Self-attentive Sentence Embeddingcited by 318
13 more
  1. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  2. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  3. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  4. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  5. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  6. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
  7. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 92
  8. 2017Multilingual Speech Recognition With A Single End-To-End Modelcited by 87
  9. 2019Transformer-based Acoustic Modeling for Hybrid Speech Recognitioncited by 86
  10. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 81
  11. 2018Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytescited by 54
  12. 2018Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speechcited by 52
  13. 2018Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modelingcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 8; 2023: 9; 2024: 0; 2025: 0; 2026: 1.

20 papers.

Sort:

  1. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  2. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  3. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 38
  4. 2021Fine-tuning wav2vec2 for speaker recognitioncited by 35
  5. 2022End-to-End Speech Translation for Code Switched Speechcited by 26
  6. 2022Streaming End-to-End Multilingual Speech Recognition with Joint Language Identificationcited by 24
  7. 2021Conformer-Based Self-Supervised Learning for Non-Speech Audio Taskscited by 22
  8. 2023Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyondcited by 10
  9. 2023Fumbling in Babel: An Investigation into ChatGPT's Language Identification Abilitycited by 6
  10. 2022Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Modelscited by 5
  11. 2023SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognitioncited by 4
  12. 2022Pretraining Approaches for Spoken Language Recognition: TalTech Submission to the OLR 2021 Challengecited by 2
8 more
  1. 2023Speaker and Language Change Detection using Wav2vec2 and Whispercited by 2
  2. 2023Synthetic Cross-accent Data Augmentation for Automatic Speech Recognitioncited by 1
  3. 2023From `Snippet-lects' to Doculects and Dialects: Leveraging Neural Representations of Speech for Placing Audio Signals in a Language Landscapecited by 1
  4. 2022Towards a Common Speech Analysis Enginecited by 0
  5. 2022Detecting Vocal Fatigue with Neural Embeddingscited by 0
  6. 2023Improving Spoken Language Identification with Map-Mixcited by 0
  7. 2023LaDA: Latent Dialogue Action For Zero-shot Cross-lingual Neural Network Language Modelingcited by 0
  8. 2026Spectrogram features for audio and speech analysiscited by 0