mlsanityData to 14 Oct 2026

When Large Language Models Meet Speech: A Survey on Integration Approaches

arXiv 2502.19548 ↗First posted 26 Feb 2025cs.CL cs.SD eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large language models (LLMs) have spurred interest in expanding their application beyond text-based tasks. A large number of studies have explored integrating other modalities with LLMs, notably speech modality, which is naturally related to text. This paper surveys the integration of speech with LLMs, categorizing the methodologies into three primary approaches: text-based, latent-representation-based, and audio-token-based integration. We also demonstrate how these methods are applied across various speech-related applications and highlight the challenges in this field to offer inspiration for

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 83 of its 125 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

83 of 125 references.

References · 83

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
38 more
  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  4. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  6. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  7. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  8. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  9. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  11. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  12. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  13. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  14. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  15. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  16. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  17. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  18. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  19. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  20. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  21. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  22. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  23. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  24. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  25. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  26. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  27. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  28. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  29. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  30. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  31. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  32. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  33. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  34. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  35. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 72
  36. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  37. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  38. 2023What Do Self-Supervised Speech Models Know About Words?cited by 64

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2026Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gapcited by 7
  2. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  3. 2025Conversational Speech Reveals Structural Robustness Failures in SpeechLLM Backbonescited by 2
  4. 2026PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialectscited by 1
  5. 2026ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generationcited by 0
  6. 2026UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perceptioncited by 0
  7. 2026AOR-Bench: Do Large Audio Language Models Over-Refuse Pseudo-Harmful Queries?cited by 0
  8. 2026NouveauVoice: Generating Novel Pseudo Speakers for Voice Anonymizationcited by 0
  9. 2026A Cross Community Agenda for Speech AIcited by 0