mlsanityData to 7 Oct 2026

Recent Advances in Speech Language Models: A Survey

arXiv 2410.03751 ↗First posted 1 Oct 2024cs.CL cs.SD eess.AScited by 85

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have recently garnered significant attention, primarily for their capabilities in text-based interactions. However, natural human interaction often relies on speech, necessitating a shift towards voice-based models. A straightforward approach to achieve this involves a pipeline of ``Automatic Speech Recognition (ASR) + LLM + Text-to-Speech (TTS)", where input speech is transcribed to text, processed by an LLM, and then converted back to speech. Despite being straightforward, this method suffers from inherent limitations, such as information loss during modality conversion, significant latency due to the complex pipeline, and error accumulation across the three stages. To address these issues, Speech Language Models (SpeechLMs) -- end-to-end models that generate speech without converting from text -- have emerged as a promising alternative. This survey paper provides the first comprehensive overview of recent methodologies for constructing SpeechLMs, detailing the key components of their architecture and the various training recipes integral to their development. Additionally, we systematically survey the various capabilities of SpeechLMs, categorize their evaluation metrics, and discuss the challenges and future research directions in this rapidly evolving field. The GitHub repository is available at https://github.com/dreamtheater123/Awesome-SpeechLM-Survey

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 138 of its 183 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

138 of 183 references.

References · 138

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2017Neural Discrete Representation Learningcited by 4,650
38 more
  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  10. 2024Qwen2 Technical Reportcited by 2,076
  11. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  12. 2024Mixtral of Expertscited by 1,785
  13. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  14. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  15. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  16. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  17. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  18. 2023PaLM 2 Technical Reportcited by 1,154
  19. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  20. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  21. 2023A Survey on Multimodal Large Language Modelscited by 898
  22. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  23. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  24. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  25. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  26. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  27. 2022High Fidelity Neural Audio Compressioncited by 673
  28. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  29. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  30. 2021AST: Audio Spectrogram Transformercited by 634
  31. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  32. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  33. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  34. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  35. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  36. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  37. 2024Large Language Models: A Surveycited by 506
  38. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492

50 most cited of 138 All 138 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 44; 2026: 39.

85 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  2. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  3. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  4. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  5. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 30
  6. 2025Speech DF Arena: A Leaderboard for Speech DeepFake Detection Modelscited by 20
  7. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interactioncited by 18
  8. 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Modelscited by 16
  9. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 16
  10. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 13
  11. 2025Game-Time: Evaluating Temporal Dynamics in Spoken Language Modelscited by 13
  12. 2025Chronological Thinking in Full-Duplex Spoken Dialogue Language Modelscited by 13
38 more
  1. 2025Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problemscited by 12
  2. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 11
  3. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  4. 2025Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoningcited by 10
  5. 2025EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Modelscited by 10
  6. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 9
  7. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  8. 2025OpusLM: A Family of Open Unified Speech Language Modelscited by 9
  9. 2025VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructionscited by 9
  10. 2025From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Modelscited by 9
  11. 2025Mind-Paced Speaking: A Dual-Brain Approach to Real-Time Reasoning in Spoken Language Modelscited by 9
  12. 2024What Do Speech Foundation Models Not Learn About Speech?cited by 8
  13. 2024Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspectivecited by 8
  14. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 8
  15. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  16. 2025Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Modelcited by 7
  17. 2025`From Prompt to Perturbation': An Adaptive Framework for Voice-Based Jailbreaks on Audio LLMscited by 6
  18. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  19. 2026Anatomy of the Modality Gap: Dissecting the Internal States of End-to-End Speech LLMscited by 6
  20. 2025Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLMcited by 5
  21. 2026The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoningcited by 5
  22. 2025Speechless: Speech Instruction Training Without Speech for Low Resource Languagescited by 4
  23. 2026Do Compact SSL Backbones Matter for Audio Deepfake Detection? A Controlled Study with RAPTORcited by 4
  24. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  25. 2025SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translationcited by 3
  26. 2025Intelligibility of Text-to-Speech Systems for Mathematical Expressionscited by 3
  27. 2025GR-LLMs: Recent Advances in Generative Recommendation Based on Large Language Modelscited by 3
  28. 2025Empathy Omni: Enabling Empathetic Speech Response Generation through Large Language Modelscited by 3
  29. 2026STRUCTSURVEY: Structured Agentic Retrieval for Automated Survey Paper Generationcited by 3
  30. 2025Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Researchcited by 2
  31. 2025Conversational Speech Reveals Structural Robustness Failures in SpeechLLM Backbonescited by 2
  32. 2025UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Modelscited by 2
  33. 2026Rethinking Discrete Speech Representation Tokens for Accent Generationcited by 2
  34. 2026Spatial Audio Question Answering and Reasoning on Dynamic Source Movementscited by 2
  35. 2026VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMscited by 2
  36. 2026The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMscited by 2
  37. 2026Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLMcited by 2
  38. 2026Towards Automated Air Traffic Safety Assessment Around Non-Towered Airports Using Large Language Modelscited by 2

50 most cited of 85 All 85 (JSON)