mlsanityData to 7 Oct 2026

Retrieval Augmented End-to-End Spoken Dialog Models

arXiv 2402.01828 ↗First posted 2 Feb 2024cs.CL cs.AI cs.SDcited by 20

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

We recently developed SLM, a joint speech and language model, which fuses a pretrained foundational speech model and a large language model (LLM), while preserving the in-context learning capability intrinsic to the pretrained LLM. In this paper, we apply SLM to speech dialog applications where the dialog states are inferred directly from the audio signal. Task-oriented dialogs often contain domain-specific entities, i.e., restaurants, hotels, train stations, and city names, which are difficult to recognize, however, critical for the downstream applications. Inspired by the RAG (retrieval-augmented generation) paradigm, we propose a retrieval augmented SLM (ReSLM) that overcomes this weakness. We first train a speech retriever to retrieve text entities mentioned in the audio. The retrieved entities are then added as text inputs to the underlying SLM to bias model predictions. We evaluated ReSLM on speech MultiWoz task (DSTC-11 challenge), and found that this retrieval augmentation boosts model performance, achieving joint goal accuracy (38.6% vs 32.7%), slot error rate (20.6% vs 24.8%) and ASR word error rate (5.5% vs 6.7%). While demonstrated on dialog state tracking, our approach is broadly applicable to other speech tasks requiring contextual information or domain-specific entities, such as contextual ASR with biasing capability.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 37 of its 47 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

37 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Retrieval-Augmented Text-to-Audio Generation14 Sep

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
24 more
  1. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  3. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  4. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  5. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  6. 2019Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systemscited by 215
  7. 2020Nearest Neighbor Machine Translationcited by 209
  8. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  9. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  10. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  11. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  12. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  13. 2022Description-Driven Task-Oriented Dialog Modelingcited by 50
  14. 2019Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answeringcited by 39
  15. 2021Controllable Semantic Parsing via Retrieval Augmentationcited by 35
  16. 2018Dialog-context aware end-to-end speech recognitioncited by 27
  17. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  18. 2018Personalized Language Model for Query Auto-Completioncited by 25
  19. 2022Integrating Text Inputs For Training and Adapting RNN Transducer ASR Modelscited by 20
  20. 2022Accelerating RNN-T Training and Inference Using CTC guidancecited by 17
  21. 2022Show, Don't Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialoguecited by 16
  22. 2018Contextual Language Model Adaptation for Conversational Agentscited by 10
  23. 2017Dialog Context Language Modeling with Recurrent Neural Networkscited by 6
  24. 2023Using External Off-Policy Speech-To-Text Mappings in Contextual End-To-End Automated Speech Recognitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 9; 2026: 5.

20 papers.

Sort:

By area:

  1. 2024Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generationcited by 25
  2. 2024FunnelRAG: A Coarse-to-Fine Progressive Retrieval Paradigm for RAGcited by 18
  3. 2025SHANKS: Simultaneous Hearing and Thinking for Spoken Language Modelscited by 16
  4. 2024Speech Retrieval-Augmented Generation without Automatic Speech Recognitioncited by 14
  5. 2025Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generationcited by 5
  6. 2024LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generationcited by 4
  7. 2025Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMscited by 4
  8. 2024Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learningcited by 2
  9. 2024On-Device LLMs for SMEs: Challenges and Opportunitiescited by 2
  10. 2025NoteIt: A System Converting Instructional Videos to Interactable Notes Through Multimodal Video Understandingcited by 2
  11. 2025End-to-end Contrastive Language-Speech Pretraining Model For Long-form Spoken Question Answeringcited by 2
  12. 2026Knowledge Is Not Static: Order-Aware Hypergraph RAG for Language Modelscited by 2
8 more
  1. 2025DACL-RAG: Data Augmentation Strategy with Curriculum Learning for Retrieval-Augmented Generationcited by 1
  2. 2025Joint Speech and Text Training for LLM-Based End-to-End Spoken Dialogue State Trackingcited by 1
  3. 2026MedSpeak: A Knowledge Graph-Aided ASR Error Correction Framework for Spoken Medical QAcited by 1
  4. 2026From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decodingcited by 1
  5. 2025What Do Humans Hear When Interacting? Experiments on Selective Listening for Evaluating ASR of Spoken Dialogue Systemscited by 0
  6. 2025Expressive Speech Retrieval using Natural Language Descriptions of Speaking Stylecited by 0
  7. 2026S2Dialog: Multimodal Dialogue Retrieval with Semantic and Acoustic-Style Modelingcited by 0
  8. 2026Don't Just Listen, Try Planning: Graph-based Retrieval-Generation Agent for Long-form Audio Meeting Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding
  2. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing
  3. 2022AARGH! End-to-end Retrieval-Generation for Task-Oriented Dialog
  4. 2026RASST: Retrieval-Augmented Simultaneous Speech Translation
  5. 2025WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models
  6. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues
  7. 2025SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation
  8. 2025Retrieval-Augmented Speech Recognition Approach for Domain Challenges