mlsanityData to 14 Oct 2026

Merging the Knowledge of LLMs for Automatic Speech Recognition

arXiv 2609.15743 ↗First posted 14 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Automatic speech recognition (ASR) systems, trained on paired speech-text data, have been improved by leveraging language models (LMs) trained on text-only data. LM fusion methods such as shallow fusion and density ratio are well-established methods that incorporate external LMs during ASR decoding. However, they incur additional computational costs due to LM inference, which is particularly problematic for recent larger LMs. In this study, we propose incorporating external LMs via model merging. This method integrates the LMs directly into the parameters of an LLM-based ASR model, requiring no additional computational cost at inference. We formulate domain extension and transfer via arithmetic operations on LoRA parameters. Experimental evaluations were conducted for the domain adaptation of LLM-based ASR trained on CSJ and LibriSpeech. We show that our LM merging consistently improved the ASR performance in the target domains, without degrading inference speed or memory footprint.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 49 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

42 of 49 references.

References · 42

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  8. 2022Editing Models with Task Arithmeticcited by 1,120
  9. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  10. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  11. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  12. 2025Qwen3-Omni Technical Reportcited by 497
30 more
  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2025Kimi-Audio Technical Reportcited by 222
  3. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  4. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  5. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  6. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  7. 2023End-to-End Speech Recognition: A Surveycited by 129
  8. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 111
  9. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  10. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
  11. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 78
  12. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  13. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  14. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 72
  15. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 59
  16. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  17. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 42
  18. 2020Distilling the Knowledge of BERT for Sequence-to-Sequence ASRcited by 41
  19. 2024LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMscited by 33
  20. 2025Granite-speech: open-source speech-aware LLMs with strong English ASR capabilitiescited by 27
  21. 2025OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaningcited by 27
  22. 2023Large-scale Language Model Rescoring on Long-form Datacited by 14
  23. 2024Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognitioncited by 14
  24. 2023Massively Multilingual Shallow Fusion with Large Language Modelscited by 10
  25. 2026Transport and Merge: Cross-Architecture Merging for Large Language Modelscited by 7
  26. 2022Non-autoregressive Error Correction for CTC-based ASR with Phone-conditioned Masked LMcited by 6
  27. 2024Task Arithmetic for Language Expansion in Speech Translationcited by 6
  28. 2025Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognitioncited by 5
  29. 2024Applying LLMs for Rescoring N-best ASR Hypotheses of Casual Conversations: Effects of Domain Adaptation and Context Carry-overcited by 4
  30. 2025Rare Word Recognition and Translation Without Fine-Tuning via Task Vector in Speech Modelscited by 1

Built on by

Not cited yet.