mlsanityData to 14 Oct 2026

Scaling Analysis of Interleaved Speech-Text Language Models

arXiv 2504.02398 ↗First posted 3 Apr 2025cs.CL cs.SD eess.AScited by 14

Not on the map. Why?

Abstract (arXiv)

Existing Speech Language Model (SLM) scaling analysis paints a bleak picture. It predicts that SLMs require much more compute and data compared to text, leading some to question the feasibility of training high-quality SLMs. However, modern SLMs are often initialised from pre-trained TextLMs using speech-text interleaving to allow knowledge transfer. This raises the question - "Do interleaved SLMs scale more efficiently than textless-SLMs?" In this paper we answer a resounding yes! We conduct scaling analysis of interleaved SLMs by training several dozen and analysing the scaling trends. We see that under this setup SLMs scale more efficiently with compute. Additionally, our results indicate that the scaling dynamics significantly differ from textless-SLMs, suggesting one should allocate notably more of the compute budget to increasing model size over training tokens. We also study the role of synthetic data and TextLM model families in unlocking this potential. Results suggest that our scaled up model achieves comparable semantic speech performance to leading models, while using less compute and data. We open source models, samples, and data - https://pages.cs.huji.ac.il/adiyoss-lab/sims/ .

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 55 of its 63 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

55 of 63 references.

References · 55

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  12. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
38 more
  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  2. 2022High Fidelity Neural Audio Compressioncited by 687
  3. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  4. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  5. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  6. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  7. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  8. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  10. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  11. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  12. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  13. 2023Scaling Data-Constrained Language Modelscited by 403
  14. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  15. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  16. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  17. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  18. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  19. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  20. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  21. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  22. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 101
  23. 2023Textually Pretrained Speech Language Modelscited by 101
  24. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  25. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  26. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  27. 2024Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Taskscited by 71
  28. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  29. 2023Analysing Discrete Self Supervised Speech Representation for Spoken Language Modelingcited by 47
  30. 2024Scaling Speech-Text Pre-training with Synthetic Interleaved Datacited by 47
  31. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 44
  32. 2024Scaling Properties of Speech Language Modelscited by 32
  33. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  34. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 27
  35. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  36. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  37. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 23
  38. 2025OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Modelscited by 17

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

By area:

  1. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 17
  2. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  3. 2025StressTest: Can YOUR Speech LM Handle the Stress?cited by 7
  4. 2026Covo-Audio Technical Reportcited by 7
  5. 2026Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokenscited by 4
  6. 2025ALAS: An Automatic Latent Alignment Score for Audio Language Modelscited by 3
  7. 2025Scaling Spoken Language Models with Syllabic Speech Tokenizationcited by 2
  8. 2025Acoustic-based Gender Differentiation in Speech-aware Language Modelscited by 1
  9. 2026Scaling Properties of Continuous Diffusion Spoken Language Modelscited by 1
  10. 2026Quantifying the Generation Modality Gap in Speech-Text Language Modelscited by 1
  11. 2026Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignmentcited by 0
  12. 2026Interleaved Speech Language Models Latently Work In Textcited by 0
2 more
  1. 2026StreamAlign: Streaming Text-Aligned Speech Tokenizationcited by 0
  2. 2026Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictionscited by 0