mlsanityData to 7 Oct 2026

SALSA: Speedy ASR-LLM Synchronous Aggregation

arXiv 2408.16542 ↗First posted 29 Aug 2024cs.CL cs.LG cs.SDcited by 8

Not on the map. Why?

Abstract (arXiv)

Harnessing pre-trained LLMs to improve ASR systems, particularly for low-resource languages, is now an emerging area of research. Existing methods range from using LLMs for ASR error correction to tightly coupled systems that replace the ASR decoder with the LLM. These approaches either increase decoding time or require expensive training of the cross-attention layers. We propose SALSA, which couples the decoder layers of the ASR to the LLM decoder, while synchronously advancing both decoders. Such coupling is performed with a simple projection of the last decoder state, and is thus significantly more training efficient than earlier approaches. A challenge of our proposed coupling is handling the mismatch between the tokenizers of the LLM and ASR systems. We handle this mismatch using cascading tokenization with respect to the LLM and ASR vocabularies. We evaluate SALSA on 8 low-resource languages in the FLEURS benchmark, yielding substantial WER reductions of up to 38%.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

24 of 39 references.

References · 24

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  5. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  6. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  7. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  8. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  9. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  10. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 109
  11. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 86
  12. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
12 more
  1. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  2. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  3. 2023Can Generative Large Language Models Perform ASR Error Correction?cited by 56
  4. 2023LLaSM: Large Language and Speech Modelcited by 56
  5. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  6. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 41
  7. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 30
  8. 2022Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systemscited by 23
  9. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  10. 2021Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognitioncited by 19
  11. 2023Leveraging Large Language Models for Exploiting ASR Uncertaintycited by 15
  12. 2023Instruction-Following Speech Recognitioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 3.

8 papers.

  1. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  2. 2025Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognitioncited by 9
  3. 2025Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languagescited by 7
  4. 2025Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognitioncited by 5
  5. 2024Tiny-Align: Bridging Automatic Speech Recognition and Large Language Model on the Edgecited by 1
  6. 2026MEUSLI: a Multilingual Projector for LLM-based ASR and Beyondcited by 1
  7. 2026Measuring the Redundancy of Decoder Layers in SpeechLLMscited by 0
  8. 2026Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactionscited by 0