mlsanityData to 14 Oct 2026

Distilling a Pretrained Language Model to a Multilingual ASR Model

arXiv 2206.12638 ↗First posted 25 Jun 2022cs.CL cs.AI cs.SDcited by 7

Not on the map. Why?

Abstract (arXiv)

Multilingual speech data often suffer from long-tailed language distribution, resulting in performance degradation. However, multilingual text data is much easier to obtain, yielding a more useful general language model. Hence, we are motivated to distill the rich knowledge embedded inside a well-trained teacher text model to the student speech model. We propose a novel method called the Distilling a Language model to a Speech model (Distill-L2S), which aligns the latent representations of two different modalities. The subtle differences are handled by the shrinking mechanism, nearest-neighbor interpolation, and a learnable linear projection layer. We demonstrate the effectiveness of our distillation method by applying it to the multilingual automatic speech recognition (ASR) task. We distill the transformer-based cross-lingual language model (InfoXLM) while fine-tuning the large-scale multilingual ASR model (XLSR-wav2vec 2.0) for each language. We show the superiority of our method on 20 low-resource languages of the CommonVoice dataset with less than 100 hours of speech data.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 25 Jun 2022.

Builds on

32 of 46 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  10. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  12. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
20 more
  1. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  2. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  3. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  4. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  5. 2021Unsupervised Speech Recognitioncited by 148
  6. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  7. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 93
  8. 2017Multilingual Speech Recognition With A Single End-To-End Modelcited by 87
  9. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  10. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  11. 2017Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Modelcited by 37
  12. 2020Adapt-and-Adjust: Overcoming the Long-Tail Problem of Multilingual Speech Recognitioncited by 35
  13. 2018Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spacescited by 34
  14. 2021Wav2vec-C: A Self-supervised Model for Speech Representation Learningcited by 25
  15. 2018Sequence-based Multi-lingual Low Resource Speech Recognitioncited by 20
  16. 2019One-To-Many Multilingual End-to-end Speech Translationcited by 17
  17. 2021Temporal Knowledge Distillation for On-device Audio Classificationcited by 15
  18. 2015Very Deep Multilingual Convolutional Neural Networks for LVCSRcited by 11
  19. 2020Speech to Text Adaptation: Towards an Efficient Cross-Modal Distillationcited by 11
  20. 2020Two-stage Textual Knowledge Distillation for End-to-End Spoken Language Understandingcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 2; 2026: 2.

7 papers.

  1. 2024Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASRcited by 3
  2. 2023Comparison of L2 Korean pronunciation error patterns from five L1 backgrounds by using automatic phonetic transcriptioncited by 1
  3. 2025Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASRcited by 1
  4. 2024CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generationcited by 0
  5. 2025New Insights into Optimal Alignment of Acoustic and Linguistic Representations for Knowledge Transfer in ASRcited by 0
  6. 2026Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languagescited by 0
  7. 2026Optimal transport meets speech: a tutorial reviewcited by 0