mlsanityData to 7 Oct 2026

Scaling Properties of Continuous Diffusion Spoken Language Models

arXiv 2604.24416 ↗First posted 27 Apr 2026cs.CL cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Speech-only spoken language models (SLMs) lag behind text and text-speech models in performance, with recent discrete autoregressive (AR) SLMs indicating significant computational and data demands to match text models. Since discretizing continuous speech for AR creates bottlenecks, we explore whether continuous diffusion (CD) SLM is more viable. To quantify the SLMs linguistic quality, we introduce the phoneme Jensen-Shannon divergence (pJSD) metric. Our analysis reveals CD SLMs, mirroring AR behavior, exhibit scaling laws for validation loss and pJSD, and show optimal token-to-parameter ratios decreasing as compute scales. However, for the latter, loss becomes insensitive to choice of data and model sizes, showing potential for fast inference. Scaling CD SLMs to 16B parameters with tens of millions of hours of conversational data enables generation of emotive, prosodic, multi-speaker, multilingual speech, though achieving long-form coherence remains a significant challenge.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 73 of its 85 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

73 of 85 references.

References · 73

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
38 more
  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  4. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2022Diffusion Models in Vision: A Surveycited by 1,008
  7. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  8. 2021Scaling Vision Transformerscited by 993
  9. 2025Large Language Diffusion Modelscited by 872
  10. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  11. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  13. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  14. 2023Scaling Data-Constrained Language Modelscited by 391
  15. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  16. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  17. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 320
  18. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  19. 2024Guiding a Diffusion Model with a Bad Version of Itselfcited by 286
  20. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  21. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 242
  22. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 235
  23. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  24. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 189
  25. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  26. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 165
  27. 2025Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200kcited by 157
  28. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speechcited by 156
  29. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 149
  30. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speechcited by 147
  31. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  32. 2025Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Soundcited by 129
  33. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  34. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  35. 2023Textually Pretrained Speech Language Modelscited by 99
  36. 2024Long-form music generation with latent diffusioncited by 88
  37. 2020Universal Phone Recognition with a Multilingual Allophone Systemcited by 81
  38. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 70

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Quantifying the Generation Modality Gap in Speech-Text Language Modelscited by 0