mlsanityData to 7 Oct 2026

Multi-Modal Retrieval For Large Language Model Based Speech Recognition

arXiv 2406.09618 ↗First posted 13 Jun 2024cs.CL cs.AI cs.IRcited by 4

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Retrieval is a widely adopted approach for improving language models leveraging external information. As the field moves towards multi-modal large language models, it is important to extend the pure text based methods to incorporate other modalities in retrieval as well for applications across the wide spectrum of machine learning tasks and data types. In this work, we propose multi-modal retrieval with two approaches: kNN-LM and cross-attention techniques. We demonstrate the effectiveness of our retrieval approaches empirically by applying them to automatic speech recognition tasks with access to external information. Under this setting, we show that speech-based multi-modal retrieval outperforms text based retrieval, and yields up to 50 % improvement in word error rate over the multi-modal language model baseline. Furthermore, we achieve state-of-the-art recognition results on the Spoken-Squad question answering dataset.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 39 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

31 of 39 references.

References · 31

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2020On Layer Normalization in the Transformer Architecturecited by 906
  9. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  10. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  11. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  12. 2023Augmented Language Models: a Surveycited by 398
19 more
  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  2. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  3. 2022Training Language Models with Memory Augmentationcited by 112
  4. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 98
  5. 2018Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehensioncited by 74
  6. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  7. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 71
  8. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  9. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 70
  10. 2020Deep Shallow Fusion for RNN-T Personalizationcited by 64
  11. 2020Memformer: A Memory-Augmented Transformer for Sequence Modelingcited by 63
  12. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Studycited by 63
  13. 2022Contextual Adapters for Personalized Speech Recognition in Neural Transducerscited by 57
  14. 2021Context-Aware Transformer Transducer for Speech Recognitioncited by 56
  15. 2021Personalization Strategies for End-to-End Speech Recognition Systemscited by 26
  16. 2021Domain-aware Neural Language Models for Speech Recognitioncited by 13
  17. 2023Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understandingcited by 9
  18. 2023FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Modelcited by 7
  19. 2023On-the-fly Text Retrieval for End-to-End ASR Adaptationcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agentcited by 14
  2. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answeringcited by 10
  3. 2025NoLoCo: No-all-reduce Low Communication Training Method for Large Modelscited by 4
  4. 2024Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrievalcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems
  2. 2024Retrieval Augmented End-to-End Spoken Dialog Models
  3. 2024Speech Recognition Rescoring with Large Speech-Text Foundation Models
  4. 2025Retrieval-Augmented Speech Recognition Approach for Domain Challenges
  5. 2025Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval
  6. 2023MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval
  7. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
  8. 2025RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering