mlsanityData to 7 Oct 2026

Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding

arXiv 2306.07944 ↗First posted 8 Jun 2023eess.AS cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have been applied in the speech domain, often incurring a performance drop due to misaligned between speech and language representations. To bridge this gap, we propose a joint speech and language model (SLM) using a Speech2Text adapter, which maps speech into text token embedding space without speech information loss. Additionally, using a CTC-based blank-filtering, we can reduce the speech sequence length to that of text. In speech MultiWoz dataset (DSTC11 challenge), SLM largely improves the dialog state tracking (DST) performance (24.7% to 28.4% accuracy). Further to address errors on rare entities, we augment SLM with a Speech2Entity retriever, which uses speech to retrieve relevant entities, and then adds them to the original SLM input as a prefix. With this retrieval-augmented SLM (ReSLM), the DST performance jumps to 34.6% accuracy. Moreover, augmenting the ASR task with the dialog understanding task improves the ASR performance from 9.4% to 8.5% WER.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 27 of its 35 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

27 of 35 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  8. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  9. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  10. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  11. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  12. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
15 more
  1. 2019Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systemscited by 215
  2. 2020Nearest Neighbor Machine Translationcited by 209
  3. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 98
  4. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  5. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  6. 2019Speech Recognition with Augmented Synthesized Speechcited by 60
  7. 2022Description-Driven Task-Oriented Dialog Modelingcited by 50
  8. 2019Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answeringcited by 39
  9. 2021Controllable Semantic Parsing via Retrieval Augmentationcited by 35
  10. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  11. 2022Integrating Text Inputs For Training and Adapting RNN Transducer ASR Modelscited by 20
  12. 2022Accelerating RNN-T Training and Inference Using CTC guidancecited by 17
  13. 2022Show, Don't Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialoguecited by 16
  14. 2022Speech Aware Dialog System Technology Challenge (DSTC11)cited by 10
  15. 2022Knowledge-grounded Dialog State Trackingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 1; 2026: 1.

9 papers.

  1. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  2. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  3. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  4. 2024BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillationcited by 8
  5. 2024Multi-Modal Retrieval For Large Language Model Based Speech Recognitioncited by 4
  6. 2024Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrievalcited by 3
  7. 2024Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learningcited by 2
  8. 2025SparQLe: Speech Queries to Text Translation Through LLMscited by 1
  9. 2026Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Retrieval Augmented End-to-End Spoken Dialog Models
  2. 2025SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation
  3. 2024Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems
  4. 2024Speech Recognition Rescoring with Large Speech-Text Foundation Models
  5. 2026Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving
  6. 2024Speech Retrieval-Augmented Generation without Automatic Speech Recognition
  7. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data
  8. 2024Retrieval Augmented Correction of Named Entity Speech Recognition Errors