mlsanityData to 14 Oct 2026

Connecting Speech Encoder and Large Language Model for ASR

arXiv 2309.13963 ↗First posted 25 Sep 2023eess.AS cs.CL cs.SDcited by 80

Not on the map. Why?

Abstract (arXiv)

The impressive capability and versatility of large language models (LLMs) have aroused increasing attention in automatic speech recognition (ASR), with several pioneering studies attempting to build integrated ASR models by connecting a speech encoder with an LLM. This paper presents a comparative study of three commonly used structures as connectors, including fully connected layers, multi-head cross-attention, and Q-Former. Speech encoders from the Whisper model series as well as LLMs from the Vicuna model series with different model sizes were studied. Experiments were performed on the commonly used LibriSpeech, Common Voice, and GigaSpeech datasets, where the LLMs with Q-Formers demonstrated consistent and considerable word error rate (WER) reductions over LLMs with other connector structures. Q-Former-based LLMs can generalise well to out-of-domain datasets, where 12% relative WER reductions over the Whisper baseline ASR model were achieved on the Eval2000 test set without using any in-domain training data from Switchboard. Moreover, a novel segment-level Q-Former is proposed to enable LLMs to recognise speech segments with a duration exceeding the limitation of the encoders, which results in 17% relative WER reductions over other connector structures on 90-second-long speech data.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Builds on 35 of its 38 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

35 of 38 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  12. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
23 more
  1. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  2. 2023PaLM 2 Technical Reportcited by 1,162
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  5. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  6. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  7. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  8. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  9. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  10. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  11. 2023Listen, Think, and Understandcited by 209
  12. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  13. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  14. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  15. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  16. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  17. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  18. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  19. 2023LLaSM: Large Language and Speech Modelcited by 58
  20. 2023Can Generative Large Language Models Perform ASR Error Correction?cited by 57
  21. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  22. 2023Leveraging Large Language Models for Exploiting ASR Uncertaintycited by 15
  23. 2022Random Utterance Concatenation Based Data Augmentation for Improving Short-video Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 33; 2025: 27; 2026: 14.

80 papers.

Sort:

By area:

  1. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  2. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  3. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  4. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 117
  5. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  6. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  7. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 76
  8. 2024Language Model Can Listen While Speakingcited by 63
  9. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 62
  10. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  11. 2025VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulationcited by 51
  12. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 42
38 more
  1. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  2. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 31
  3. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  4. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Modelscited by 21
  5. 2024Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learningcited by 20
  6. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 19
  7. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  8. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
  9. 2024SSR: Alignment-Aware Modality Connector for Speech Language Modelscited by 16
  10. 2025SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuningcited by 14
  11. 2024Enabling Auditory Large Language Models for Automatic Speech Quality Evaluationcited by 13
  12. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  13. 2023Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognitioncited by 10
  14. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10
  15. 2024Contextualization of ASR with LLM using phonetic retrieval-based augmentationcited by 10
  16. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  17. 2023UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructionscited by 9
  18. 2024Investigating Decoder-only Large Language Models for Speech-to-text Translationcited by 9
  19. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  20. 2025Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognitioncited by 9
  21. 2024Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimizationcited by 8
  22. 2024What Do Speech Foundation Models Not Learn About Speech?cited by 8
  23. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 8
  24. 2026Text-only adaptation in LLM-based ASR through text denoisingcited by 8
  25. 2024Using Large Language Model for End-to-End Chinese ASR and NERcited by 6
  26. 2025Universal Acoustic Adversarial Attacks for Flexible Control of Speech-LLMscited by 6
  27. 2025Can LLMs Understand Unvoiced Speech? Exploring EMG-to-Text Conversion with LLMscited by 6
  28. 2025Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysiscited by 5
  29. 2026TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Groundingcited by 5
  30. 2024An End-to-End Speech Summarization Using Large Language Modelcited by 4
  31. 2024LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generationcited by 4
  32. 2024How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Notcited by 4
  33. 2024Self-Powered LLM Modality Expansion for Large Speech-Text Modelscited by 4
  34. 2025LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriorscited by 4
  35. 2025Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMscited by 4
  36. 2025EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognitioncited by 4
  37. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  38. 2024LAMA-UT: Language Agnostic Multilingual ASR through Orthography Unification and Language-Specific Transliterationcited by 3

50 most cited of 80 All 80 (JSON)