mlsanityData to 14 Oct 2026

Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study

arXiv 2401.12789 ↗First posted 23 Jan 2024cs.CL cs.SD eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

In the era of large models, the autoregressive nature of decoding often results in latency serving as a significant bottleneck. We propose a non-autoregressive LM-fused ASR system that effectively leverages the parallelization capabilities of accelerator hardware. Our approach combines the Universal Speech Model (USM) and the PaLM 2 language model in per-segment scoring mode, achieving an average relative WER improvement across all languages of 10.8% on FLEURS and 3.6% on YouTube captioning. Furthermore, our comprehensive ablation study analyzes key parameters such as LLM size, context length, vocabulary size, fusion methodology. For instance, we explore the impact of LLM size ranging from 128M to 340B parameters on ASR performance. This study provides valuable insights into the factors influencing the effectiveness of practical large-scale LM-fused speech recognition systems.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 29 of its 32 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

29 of 32 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  11. 2023PaLM 2 Technical Reportcited by 1,162
  12. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
17 more
  1. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  2. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 150
  3. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 111
  4. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  5. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  6. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  7. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  8. 2019A comparison of end-to-end models for long-form speech recognitioncited by 42
  9. 2021Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERTcited by 31
  10. 2023Improving Massively Multilingual ASR With Auxiliary CTC Objectivescited by 31
  11. 2020RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutionscited by 26
  12. 2022E2E Segmenter: Joint Segmenting and Decoding for Long-Form ASRcited by 17
  13. 2023Large-scale Language Model Rescoring on Long-form Datacited by 14
  14. 2021Non-autoregressive Transformer-based End-to-end ASR using BERTcited by 13
  15. 2023Massively Multilingual Shallow Fusion with Large Language Modelscited by 10
  16. 2022E2E Segmentation in a Two-Pass Cascaded Encoder ASR Modelcited by 3
  17. 2023Semantic Segmentation with Bidirectional Language Models Improves Long-form ASRcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 3.

12 papers.

  1. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 63
  2. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  3. 2024Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forwardcited by 17
  4. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  5. 2026Text-only adaptation in LLM-based ASR through text denoisingcited by 8
  6. 2025Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognitioncited by 5
  7. 2024Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCRcited by 4
  8. 2024LAMA-UT: Language Agnostic Multilingual ASR through Orthography Unification and Language-Specific Transliterationcited by 3
  9. 2025Drax: Speech Recognition with Discrete Flow Matchingcited by 3
  10. 2026Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projectioncited by 2
  11. 2025TokenVerse++: Towards Flexible Multitask Learning with Dynamic Task Activationcited by 0
  12. 2026Culturally-Grounded Governance for Multilingual Language Models: Rights, Data Boundaries, and Accountable AI Designcited by 0