mlsanityData to 7 Oct 2026

How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not

arXiv 2409.17044 ↗First posted 25 Sep 2024cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

The remarkable performance achieved by Large Language Models (LLM) has driven research efforts to leverage them for a wide range of tasks and input modalities. In speech-to-text (S2T) tasks, the emerging solution consists of projecting the output of the encoder of a Speech Foundational Model (SFM) into the LLM embedding space through an adapter module. However, no work has yet investigated how much the downstream-task performance depends on each component (SFM, adapter, LLM) nor whether the best design of the adapter depends on the chosen SFM and LLM. To fill this gap, we evaluate the combination of 5 adapter modules, 2 LLMs (Mistral and Llama), and 2 SFMs (Whisper and SeamlessM4T) on two widespread S2T tasks, namely Automatic Speech Recognition and Speech Translation. Our results demonstrate that the SFM plays a pivotal role in downstream performance, while the adapter choice has moderate impact and depends on the SFM and LLM.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

25 of 31 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2023Mistral 7Bcited by 3,065
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  9. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  10. 2024Qwen2-Audio Technical Reportcited by 543
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  12. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
13 more
  1. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  2. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  3. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 107
  4. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 86
  5. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  6. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 76
  7. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 75
  8. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  9. 2021CTC-based Compression for Direct Speech Translationcited by 45
  10. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  11. 2024LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Modelscited by 12
  12. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  13. 2024BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026MEUSLI: a Multilingual Projector for LLM-based ASR and Beyondcited by 1
  2. 2025PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMscited by 0
  3. 2026voice2mode: Phonation Mode Classification in Singing using Self-Supervised Speech Modelscited by 0
  4. 2026Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learningcited by 0