mlsanityData to 7 Oct 2026

LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generation

arXiv 2409.08597 ↗First posted 13 Sep 2024cs.SD cs.CL eess.AScited by 4

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

Recent advancements in integrating speech information into large language models (LLMs) have significantly improved automatic speech recognition (ASR) accuracy. However, existing methods often constrained by the capabilities of the speech encoders under varied acoustic conditions, such as accents. To address this, we propose LA-RAG, a novel Retrieval-Augmented Generation (RAG) paradigm for LLM-based ASR. LA-RAG leverages fine-grained token-level speech datastores and a speech-to-speech retrieval mechanism to enhance ASR accuracy via LLM in-context learning (ICL) capabilities. Experiments on Mandarin and various Chinese dialect datasets demonstrate significant improvements in ASR accuracy compared to existing methods, validating the effectiveness of our approach, especially in handling accent variations.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 31 references that are in the map. First posted on arXiv 13 Sep 2024.

Builds on

27 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Retrieval Augmented End-to-End Spoken Dialog Models2 Feb

Other references · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  9. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  10. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
  11. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  12. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 75
14 more
  1. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  2. 2023Can Generative Large Language Models Perform ASR Error Correction?cited by 56
  3. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  4. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  5. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 41
  6. 2023Can Whisper perform speech-based in-context learning?cited by 35
  7. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 30
  8. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 30
  9. 2021Learning Kernel-Smoothed Machine Translation with Retrieved Examplescited by 25
  10. 2023End-to-End Speech Recognition Contextualization with Large Language Modelscited by 22
  11. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  12. 2023kNN-CTC: Enhancing ASR via Retrieval of CTC Pseudo Labelscited by 8
  13. 2023Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Taskscited by 5
  14. 2024Speaker-Smoothed kNN Speaker Adaptation for End-to-End ASRcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directionscited by 69
  2. 2025GEC-RAG: Improving Generative Error Correction via Retrieval-Augmented Generation for Automatic Speech Recognition Systemscited by 5
  3. 2025Game-Oriented ASR Error Correction via RAG-Enhanced LLMcited by 0
  4. 2025Incorporating Error Level Noise Embedding for Improving LLM-Assisted Robustness in Persian Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RAG-Boost: Retrieval-Augmented Generation Enhanced LLM-based Speech Recognition
  2. 2026Qwen-Audio-3.0-ASR Technical Report
  3. 2026Better Retrieval, Worse Robustness: How Multi-hop RAG Amplifies Upstream ASR Errors
  4. 2024Retrieval Augmented Correction of Named Entity Speech Recognition Errors
  5. 2025Retrieval-Augmented Speech Recognition Approach for Domain Challenges
  6. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  7. 2026Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling
  8. 2026Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs