mlsanityData to 7 Oct 2026

Massively Multilingual Shallow Fusion with Large Language Models

arXiv 2302.08917 ↗First posted 17 Feb 2023cs.CL cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

While large language models (LLM) have made impressive progress in natural language processing, it remains unclear how to utilize them in improving automatic speech recognition (ASR). In this work, we propose to train a single multilingual language model (LM) for shallow fusion in multiple languages. We push the limits of the multilingual LM to cover up to 84 languages by scaling up using a mixture-of-experts LLM, i.e., generalist language model (GLaM). When the number of experts increases, GLaM dynamically selects only two at each decoding step to keep the inference computation roughly constant. We then apply GLaM to a multilingual shallow fusion task based on a state-of-the-art end-to-end model. Compared to a dense LM of similar computation during inference, GLaM reduces the WER of an English long-tail test set by 4.4% relative. In a multilingual shallow fusion task, GLaM improves 41 out of 50 languages with an average relative WER reduction of 3.85%, and a maximum reduction of 10%. Compared to the baseline model, GLaM achieves an average WER reduction of 5.53% over 43 languages.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 32 of its 35 references that are in the map. First posted on arXiv 17 Feb 2023.

Builds on

32 of 35 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  9. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  10. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  11. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  12. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
20 more
  1. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 109
  2. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  3. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  4. 2019Language Modeling with Deep Transformerscited by 96
  5. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  6. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  7. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  8. 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
  9. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  10. 2017Listening while Speaking: Speech Chain by Deep Learningcited by 46
  11. 2020A New Training Pipeline for an Improved Neural Transducercited by 34
  12. 2018Cycle-consistency training for end-to-end speech recognitioncited by 31
  13. 2020Improving Tail Performance of a Deliberation E2E ASR Model Using a Large Text Corpuscited by 24
  14. 2021Injecting Text in Self-Supervised Speech Pretrainingcited by 24
  15. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  16. 2022Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systemscited by 20
  17. 2021Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognitioncited by 19
  18. 2018Transfer learning of language-independent end-to-end ASR with language model fusioncited by 10
  19. 2022Sentence-Select: Large-Scale Language Model Data Selection for Rare-Word Speech Recognitioncited by 10
  20. 2022Improving Deliberation by Text-Only and Semi-Supervised Trainingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 1; 2026: 1.

10 papers.

  1. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASRcited by 17
  2. 2024Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Studycited by 12
  3. 2024SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognitioncited by 4
  4. 2024Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrievalcited by 3
  5. 2023Harnessing the Zero-Shot Power of Instruction-Tuned Large Language Model in End-to-End Speech Recognitioncited by 2
  6. 2023Correction Focused Language Model Training for Speech Recognitioncited by 2
  7. 2024Are Transformers in Pre-trained LM A Good ASR Encoder? An Empirical Studycited by 2
  8. 2024Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictionscited by 1
  9. 2025Customizing Speech Recognition Model with Large Language Model Feedbackcited by 1
  10. 2026Merging the Knowledge of LLMs for Automatic Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition
  2. 2026Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling
  3. 2026Token Merging for Multilingual Speech Recognition: A Systematic Study Across Model Scale and Fine-Tuning
  4. 2024A Survey on Mixture of Experts in Large Language Models
  5. 2026Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models
  6. 2024A Survey on Efficient Inference for Large Language Models
  7. 2024Pruning Multilingual Large Language Models for Multilingual Inference
  8. 2024LOLA -- An Open-Source Massively Multilingual Large Language Model