mlsanityData to 7 Oct 2026

On The Landscape of Spoken Language Models: A Comprehensive Survey

arXiv 2504.08528 ↗First posted 11 Apr 2025cs.CL cs.SD eess.AScited by 116

Not on the map. Why?

Abstract (arXiv)

The field of spoken language processing is undergoing a shift from training custom-built, task-specific models toward using and optimizing spoken language models (SLMs) which act as universal speech processing systems. This trend is similar to the progression toward universal language models that has taken place in the field of (text) natural language processing. SLMs include both "pure" language models of speech -- models of the distribution of tokenized speech sequences -- and models that combine speech encoders with text language models, often including both spoken and written input or output. Work in this area is very diverse, with a range of terminology and evaluation settings. This paper aims to contribute an improved understanding of SLMs via a unifying literature survey of recent work in the context of the evolution of the field. Our survey categorizes the work in this area by model architecture, training, and evaluation choices, and describes some key challenges and directions for future work.

Key facts. Cited by 116 papers in the map (data through 7 Oct 2026). Builds on 180 of its 225 references that are in the map. First posted on arXiv 11 Apr 2025.

Builds on

180 of 225 references.

References · 180

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
38 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  13. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  14. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  15. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  16. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  17. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  18. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  19. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  20. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  21. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  22. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  23. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  24. 2022High Fidelity Neural Audio Compressioncited by 673
  25. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  26. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  27. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  28. 2024Qwen2-Audio Technical Reportcited by 543
  29. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  30. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  31. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  32. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  33. 2023Simple and Controllable Music Generationcited by 480
  34. 2023MusicLM: Generating Music From Textcited by 442
  35. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  36. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  37. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  38. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323

50 most cited of 180 All 180 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 49; 2026: 67.

116 papers.

Sort:

By area:

  1. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  2. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  3. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  4. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
  5. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 22
  6. 2025Audio-Aware Large Language Models as Judges for Speaking Stylescited by 17
  7. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 14
  8. 2025Game-Time: Evaluating Temporal Dynamics in Spoken Language Modelscited by 13
  9. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usagecited by 12
  10. 2025Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Modelscited by 11
  11. 2026GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Taskscited by 11
  12. 2025VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructionscited by 9
38 more
  1. 2025From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Modelscited by 9
  2. 2026Multi-Faceted Interactivity Alignment in Full-Duplex Speech Modelscited by 9
  3. 2026AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answeringcited by 8
  4. 2025In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varietiescited by 7
  5. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  6. 2025Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuningcited by 7
  7. 2025VocalBench-DF: A Benchmark for Evaluating Speech LLM Robustness to Disfluencycited by 7
  8. 2025StressTest: Can YOUR Speech LM Handle the Stress?cited by 6
  9. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  10. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  11. 2025Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modelingcited by 5
  12. 2025Evaluating Emotion Recognition in Spoken Language Models on Emotionally Incongruent Speechcited by 5
  13. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5
  14. 2026ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modelingcited by 5
  15. 2026All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluationcited by 5
  16. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  17. 2025Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMscited by 4
  18. 2025Advancing Speech Understanding in Speech-Aware Language Models with GRPOcited by 4
  19. 2025When Silence Matters: The Impact of Irrelevant Audio on Text Reasoning in Large Audio-Language Modelscited by 4
  20. 2025Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systemscited by 4
  21. 2026PRiSM: Benchmarking Phone Realization in Speech Modelscited by 4
  22. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  23. 2026Raon-Speech Technical Reportcited by 4
  24. 2025ALAS: An Automatic Latent Alignment Score for Audio Language Modelscited by 3
  25. 2025AHAMask: Reliable Task Specification for Large Audio Language Models without Instructionscited by 3
  26. 2025CantoASR: Prosody-Aware ASR-LALM Collaboration for Low-Resource Cantonesecited by 3
  27. 2026VoxServe: Streaming-Centric Serving System for Speech Language Modelscited by 3
  28. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
  29. 2026Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMscited by 3
  30. 2025An Exploration of Mamba for Speech Self-Supervised Modelscited by 2
  31. 2025TELEVAL: A Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarioscited by 2
  32. 2025GOAT-SLM: A Spoken Language Model with Paralinguistic and Speaker Characteristic Awarenesscited by 2
  33. 2025Optimal Transport Regularization for Speech Text Alignment in Spoken Language Modelscited by 2
  34. 2025Conversational Speech Reveals Structural Robustness Failures in SpeechLLM Backbonescited by 2
  35. 2025Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Modelscited by 2
  36. 2025Towards Unsupervised Speech Recognition at the Syllable-Levelcited by 2
  37. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  38. 2025AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuningcited by 2

50 most cited of 116 All 116 (JSON)