mlsanityData to 7 Oct 2026

SLM: Bridge the thin gap between speech and text foundation models

arXiv 2310.00230 ↗First posted 30 Sep 2023cs.CL cs.SD eess.AScited by 67

Not on the map. Why?

Abstract (arXiv)

We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation models to maximally preserves their capabilities, and only trains a simple adapter with just 1% (156M) of the foundation models' parameters. This adaptation not only leads SLM to achieve strong performance on conventional tasks such as speech recognition (ASR) and speech translation (AST), but also introduces the novel capability of zero-shot instruction-following for more diverse tasks: given a speech input and a text instruction, SLM is able to perform unseen generation tasks including contextual biasing ASR using real-time context, dialog generation, speech continuation, and question answering, etc. Our approach demonstrates that the representational gap between pretrained speech and language models might be narrower than one would expect, and can be bridged by a simple adaptation mechanism. As a result, SLM is not only efficient to train, but also inherits strong capabilities already acquired in foundation models of different modalities.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 30 Sep 2023.

Builds on

30 of 43 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
18 more
  1. 2023PaLM 2 Technical Reportcited by 1,154
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  3. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  4. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  5. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  7. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  8. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  9. 2023Listen, Think, and Understandcited by 206
  10. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  11. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  12. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  13. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  14. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  15. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  16. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  17. 2023AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generationcited by 22
  18. 2023Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understandingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 35; 2025: 16; 2026: 14.

67 papers.

Sort:

By area:

  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  2. 2024Qwen2-Audio Technical Reportcited by 543
  3. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  4. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  5. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  6. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  7. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 35
  8. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  9. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 32
  10. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 30
  11. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 30
  12. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
38 more
  1. 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
  2. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  3. 2024Retrieval Augmented End-to-End Spoken Dialog Modelscited by 20
  4. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 18
  5. 2024Chain-of-Thought Prompting for Speech Translationcited by 17
  6. 2024SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Modelscited by 16
  7. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 14
  8. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  9. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10
  10. 2024Advancing Multi-talker ASR Performance with Large Language Modelscited by 10
  11. 2024SpeechQE: Estimating the Quality of Direct Speech Translationcited by 10
  12. 2024Instruction Data Generation and Unsupervised Adaptation for Speech Language Modelscited by 9
  13. 2024Investigating Decoder-only Large Language Models for Speech-to-text Translationcited by 9
  14. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  15. 2024BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillationcited by 8
  16. 2025Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMscited by 8
  17. 2026Text-only adaptation in LLM-based ASR through text denoisingcited by 8
  18. 2024Are LLMs Robust for Spoken Dialogues?cited by 7
  19. 2024Fast Context-Biasing for CTC and Transducer ASR models with CTC-based Word Spottercited by 7
  20. 2024Using Large Language Model for End-to-End Chinese ASR and NERcited by 6
  21. 2024SpeechCaps: Advancing Instruction-Based Universal Speech Models with Multi-Talker Speaking Style Captioningcited by 6
  22. 2024VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuningcited by 6
  23. 2024Zero-resource Speech Translation and Recognition with LLMscited by 6
  24. 2025Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generationcited by 5
  25. 2024How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Notcited by 4
  26. 2025Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparisoncited by 4
  27. 2025Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMscited by 4
  28. 2025The Eloquence team submission for task 1 of MLC-SLM challengecited by 4
  29. 2025TurboBias: Universal ASR Context-Biasing powered by GPU-accelerated Phrase-Boosting Treecited by 4
  30. 2024Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrievalcited by 3
  31. 2025Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMscited by 3
  32. 2025SpeechLLMs for Large-scale Contextualized Zero-shot Slot Fillingcited by 3
  33. 2024ASTRA: Aligning Speech and Text Representations for Asr without Samplingcited by 2
  34. 2024Factor-Conditioned Speaking-Style Captioningcited by 2
  35. 2025Retrieval-Augmented Speech Recognition Approach for Domain Challengescited by 2
  36. 2025Cloning a Conversational Voice AI Agent from Call,Recording Datasets for Telesalescited by 2
  37. 2026Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projectioncited by 2
  38. 2026Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluationcited by 2

50 most cited of 67 All 67 (JSON)