mlsanityData to 7 Oct 2026

Improving Massively Multilingual ASR With Auxiliary CTC Objectives

arXiv 2302.12829 ↗First posted 24 Feb 2023cs.CL cs.SD eess.AScited by 31

Not on the map. Why?

Abstract (arXiv)

Multilingual Automatic Speech Recognition (ASR) models have extended the usability of speech technologies to a wide variety of languages. With how many languages these models have to handle, however, a key to understanding their imbalanced performance across different languages is to examine if the model actually knows which language it should transcribe. In this paper, we introduce our work on improving performance on FLEURS, a 102-language open ASR benchmark, by conditioning the entire model on language identity (LID). We investigate techniques inspired from recent Connectionist Temporal Classification (CTC) studies to help the model handle the large number of languages, conditioning on the LID predictions of auxiliary tasks. Our experimental results demonstrate the effectiveness of our technique over standard CTC/Attention-based hybrid models. Furthermore, our state-of-the-art systems using self-supervised models with the Conformer architecture improve over the results of prior work on FLEURS by a relative 28.4% CER. Trained models and reproducible recipes are available at https://github.com/espnet/espnet/tree/master/egs2/fleurs/asr1 .

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 24 Feb 2023.

Builds on

39 of 50 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  10. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  11. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  12. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
27 more
  1. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  2. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  3. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  4. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  5. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  6. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  7. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 95
  8. 2020Universal Phone Recognition with a Multilingual Allophone Systemcited by 81
  9. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  10. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  11. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  12. 2019Massively Multilingual Adversarial Speech Recognitioncited by 28
  13. 2021A Configurable Multilingual Model is All You Need to Recognize All Languagescited by 28
  14. 2021A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generationcited by 27
  15. 2022ASR2K: Speech Recognition for Around 2000 Languages without Audiocited by 26
  16. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  17. 2019Deja-vu: Double Feature Presentation and Iterated Loss in Deep Transformer Networkscited by 24
  18. 2022Streaming End-to-End Multilingual Speech Recognition with Joint Language Identificationcited by 24
  19. 2018Hierarchical Multi Task Learning With CTCcited by 20
  20. 2021Joint Modeling of Code-Switched and Monolingual ASR via Conditional Factorizationcited by 17
  21. 2022Building African Voicescited by 14
  22. 2021Hierarchical Conditional End-to-End ASR with CTC and Multi-Granular Subword Unitscited by 13
  23. 2022Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASRcited by 13
  24. 2022Language Adaptive Cross-lingual Speech Representation Learning with Sparse Sharing Sub-networkscited by 12
  25. 2021Differentiable Allophone Graphs for Language-Universal Speech Recognitioncited by 10
  26. 2022Improving CTC-based ASR Models with Gated Interlayer Collaborationcited by 4
  27. 2022Intermediate-layer output Regularization for Attention-based Speech Recognition with Shared Decodercited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 13; 2025: 6; 2026: 3.

31 papers.

Sort:

By area:

  1. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  2. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  3. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  4. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 26
  5. 2024Learn and Don't Forget: Adding a New Language to ASR Foundation Modelscited by 16
  6. 2023A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Taskscited by 13
  7. 2024Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Studycited by 12
  8. 2025CS-FLEURS: A Massively Multilingual and Code-Switched Speech Datasetcited by 11
  9. 2023Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyondcited by 10
  10. 2023Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learningcited by 8
  11. 2024An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglementcited by 8
  12. 2024Wav2Gloss: Generating Interlinear Glossed Text from Speechcited by 8
19 more
  1. 2025Multilingual Gloss-free Sign Language Translation: Towards Building a Sign Language Foundation Modelcited by 7
  2. 2026An Empirical Recipe for Universal Phone Recognitioncited by 5
  3. 2023A New Benchmark of Aphasia Speech Recognition and Detection Based on E-Branchformer and Multi-task Learningcited by 4
  4. 2023SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognitioncited by 4
  5. 2025Large Multimodal Models for Low-Resource Languages: A Surveycited by 4
  6. 2024Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Trainingcited by 3
  7. 2024Low-resource speech recognition and dialect identification of Irish in a multi-task frameworkcited by 3
  8. 2024Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Promptingcited by 3
  9. 2024Improving Multilingual ASR in the Wild Using Simple N-best Re-rankingcited by 2
  10. 2024CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processingcited by 2
  11. 2023Weak Alignment Supervision from Hybrid Model Improves End-to-end ASRcited by 1
  12. 2024AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologiescited by 1
  13. 2025CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognitioncited by 1
  14. 2025Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTCcited by 1
  15. 2023Leveraging Language ID to Calculate Intermediate CTC Loss for Enhanced Code-Switching Speech Recognitioncited by 0
  16. 2024Nollywood: Let's Go to the Movies!cited by 0
  17. 2025Geolocation-Aware Robust Spoken Language Identificationcited by 0
  18. 2026Overcoming Decoder Inconsistencies in Whisper for Dravidian and Low-Resource Languagescited by 0
  19. 2026Improving low-resource ASR using bilingual fine-tuning with language identification: a cross-linguistic evaluationcited by 0