mlsanityData to 13 Oct 2026

Speech Retrieval-Augmented Generation without Automatic Speech Recognition

arXiv 2412.16500 ↗First posted 21 Dec 2024eess.AS cs.AI cs.CLcited by 14

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

One common approach for question answering over speech data is to first transcribe speech using automatic speech recognition (ASR) and then employ text-based retrieval-augmented generation (RAG) on the transcriptions. While this cascaded pipeline has proven effective in many practical settings, ASR errors can propagate to the retrieval and generation steps. To overcome this limitation, we introduce SpeechRAG, a novel framework designed for open-question answering over spoken data. Our proposed approach fine-tunes a pre-trained speech encoder into a speech adapter fed into a frozen large language model (LLM)--based retrieval model. By aligning the embedding spaces of text and speech, our speech retriever directly retrieves audio passages from text-based queries, leveraging the retrieval capacity of the frozen text retriever. Our retrieval experiments on spoken question answering datasets show that direct speech retrieval does not degrade over the text-based baseline, and outperforms the cascaded systems using ASR. For generation, we use a speech language model (SLM) as a generator, conditioned on audio passages rather than transcripts. Without fine-tuning of the SLM, this approach outperforms cascaded text-based models when there is high WER in the transcripts.

Key facts. Cited by 14 papers in the map (data through 13 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 25 of its 35 references that are in the map. First posted on arXiv 21 Dec 2024.

Builds on

25 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Retrieval Augmented End-to-End Spoken Dialog Models2 Feb

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,004
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,313
  7. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 713
  8. 2022MTEB: Massive Text Embedding Benchmarkcited by 630
  9. 2020Contrastive Learning with Hard Negative Samplescited by 619
  10. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 601
  11. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 456
  12. 2023Improving Text Embeddings with Large Language Modelscited by 297
12 more
  1. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 217
  2. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 108
  3. 2018Towards end-to-end spoken language understandingcited by 92
  4. 2018Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehensioncited by 74
  5. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  6. 2021Audio Retrieval with Natural Language Queriescited by 49
  7. 2023RECAP: Retrieval-Augmented Audio Captioningcited by 26
  8. 2024Evaluation of RAG Metrics for Question Answering in the Telecom Domaincited by 19
  9. 2024SpeechDPR: End-to-End Spoken Passage Retrieval for Open-Domain Spoken Question Answeringcited by 12
  10. 2019SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answeringcited by 11
  11. 2024Prompting Large Language Models with Audio for General-Purpose Speech Summarizationcited by 6
  12. 2023OLISIA: a Cascade System for Spoken Dialogue State Trackingcited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 7; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  2. 2025WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Modelscited by 18
  3. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 11
  4. 2025VoxRAG: A Step Toward Transcription-Free RAG Systems in Spoken Question Answeringcited by 3
  5. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  6. 2026MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Modelscited by 2
  7. 2025Data Quality Challenges in Retrieval-Augmented Generationcited by 1
  8. 2025Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generationcited by 1
  9. 2026MedSpeak: A Knowledge Graph-Aided ASR Error Correction Framework for Spoken Medical QAcited by 1
  10. 2026SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noisecited by 1
  11. 2026From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decodingcited by 1
  12. 2025Expressive Speech Retrieval using Natural Language Descriptions of Speaking Stylecited by 0
2 more
  1. 2026Audio Jailbreaks in Large Audio-Language Models: Taxonomy, Attack-Defense Analysis, and Cost-Aware Evaluationcited by 0
  2. 2026A Reranker for Orchestrating Heterogeneous Speech and Text Retrieverscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Retrieval-Augmented Speech Recognition Approach for Domain Challenges
  2. 2025Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation
  3. 2025Retrieval Augmented Generation based context discovery for ASR
  4. 2025RAG-Boost: Retrieval-Augmented Generation Enhanced LLM-based Speech Recognition
  5. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  6. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
  7. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems
  8. 2025VaccineRAG: Boosting Multimodal Large Language Models' Immunity to Harmful RAG Samples