mlsanityData to 7 Oct 2026

Adaptive multilingual speech recognition with pretrained models

arXiv 2205.12304 ↗First posted 24 May 2022cs.CL cs.SD eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

Multilingual speech recognition with supervised learning has achieved great results as reflected in recent research. With the development of pretraining methods on audio and text data, it is imperative to transfer the knowledge from unsupervised multilingual models to facilitate recognition, especially in many languages with limited data. Our work investigated the effectiveness of using two pretrained models for two modalities: wav2vec 2.0 for audio and MBART50 for text, together with the adaptive weight techniques to massively improve the recognition quality on the public datasets containing CommonVoice and Europarl. Overall, we noticed an 44% improvement over purely supervised learning, and more importantly, each technique provides a different reinforcement in different languages. We also explore other possibilities to potentially obtain the best model by slightly adding either depth or relative attention to the architecture.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 34 of its 37 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

34 of 37 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  11. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
22 more
  1. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  2. 2017Learning multiple visual domains with residual adapterscited by 553
  3. 2015Listen, Attend and Spellcited by 542
  4. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  5. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  6. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  7. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  8. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  9. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  10. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  11. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  12. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 92
  13. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  14. 2019Very Deep Self-Attention Networks for End-to-End Speech Recognitioncited by 46
  15. 2018Contextual Parameter Generation for Universal Neural Machine Translationcited by 40
  16. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  17. 2021Exploiting Adapters for Cross-lingual Low-resource Speech Recognitioncited by 33
  18. 2019Improving Transformer-based Speech Recognition Using Unsupervised Pre-trainingcited by 29
  19. 2019Improving sequence-to-sequence speech recognition training with on-the-fly data augmentationcited by 29
  20. 2020Relative Positional Encoding for Speech Recognition and Direct Translationcited by 23
  21. 2021Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognitioncited by 16
  22. 2021Efficient Weight factorization for Multilingual Speech Recognitioncited by 12

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 2; 2025: 2.

9 papers.

  1. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 26
  2. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20
  3. 2022Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videoscited by 18
  4. 2023Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learningcited by 9
  5. 2024Accent conversion using discrete units with parallel data synthesized from controllable accented TTScited by 5
  6. 2025Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Reviewcited by 4
  7. 2023UML: A Universal Monolingual Output Layer for Multilingual ASRcited by 3
  8. 2022Towards continually learning new languagescited by 2
  9. 2025Bayesian Low-Rank Factorization for Robust Model Adaptationcited by 1