mlsanityData to 7 Oct 2026

Scaling Properties of Speech Language Models

arXiv 2404.00685 ↗First posted 31 Mar 2024eess.AS cs.AI cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Speech Language Models (SLMs) aim to learn language from raw audio, without textual resources. Despite significant advances, our current models exhibit weak syntax and semantic abilities. However, if the scaling properties of neural language models hold for the speech modality, these abilities will improve as the amount of compute used for training increases. In this paper, we use models of this scaling behavior to estimate the scale at which our current methods will yield a SLM with the English proficiency of text-based Large Language Models (LLMs). We establish a strong correlation between pre-training loss and downstream syntactic and semantic performance in SLMs and LLMs, which results in predictable scaling of linguistic performance. We show that the linguistic performance of SLMs scales up to three orders of magnitude more slowly than that of text-based LLMs. Additionally, we study the benefits of synthetic data designed to boost semantic understanding and the effects of coarser speech tokenization.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 31 Mar 2024.

Builds on

23 of 28 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  8. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  10. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  11. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  12. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
11 more
  1. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  2. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  3. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  4. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  5. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  6. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  7. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 98
  8. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 93
  9. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  10. 2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24
  11. 2021Scaling Laws for Acoustic Modelscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 14; 2026: 11.

30 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  2. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  3. 2024Foundation Models for Music: A Surveycited by 34
  4. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  5. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 22
  6. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 14
  7. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 13
  8. 2025Aligning Multimodal Representations through an Information Bottleneckcited by 12
  9. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 11
  10. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 9
  11. 2024Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approachcited by 8
  12. 2024Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspectivecited by 8
18 more
  1. 2025IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modelingcited by 6
  2. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  3. 2026ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modelingcited by 5
  4. 2026The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASRrightarrowLLM Pipelines?cited by 5
  5. 2025Latent Speech-Text Transformercited by 4
  6. 2025LF2AR: Accounting for Layerwise Dynamics to Improve Multimodal Adaptation of Language Modelscited by 3
  7. 2026Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokenscited by 3
  8. 2025Scaling Spoken Language Models with Syllabic Speech Tokenizationcited by 2
  9. 2025An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-trainingcited by 1
  10. 2026WavSLM: Single-Stream Speech Language Modeling via WavLM Distillationcited by 1
  11. 2026AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QAcited by 1
  12. 2026Scaling Properties of Continuous Diffusion Spoken Language Modelscited by 1
  13. 2026Self-Play Pretraining with Zero Datacited by 1
  14. 2025Cross-Lingual Interleaving for Speech Language Modelscited by 0
  15. 2026Interleaved Speech Language Models Latently Work In Textcited by 0
  16. 2026On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Modelscited by 0
  17. 2026DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialoguescited by 0
  18. 2026Quantifying the Generation Modality Gap in Speech-Text Language Modelscited by 0