mlsanityData to 14 Oct 2026

Slamming: Training a Speech Language Model on One GPU in a Day

arXiv 2502.15814 ↗First posted 19 Feb 2025cs.LG cs.AI cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

We introduce Slam, a recipe for training high-quality Speech Language Models (SLMs) on a single academic GPU in 24 hours. We do so through empirical analysis of model initialisation and architecture, synthetic training data, preference optimisation with synthetic data and tweaking all other components. We empirically demonstrate that this training recipe also scales well with more compute getting results on par with leading SLMs in a fraction of the compute cost. We hope these insights will make SLM training and research more accessible. In the context of SLM scaling laws, our results far outperform predicted compute optimal performance, giving an optimistic view to SLM feasibility. See code, data, models, samples at - https://pages.cs.huji.ac.il/adiyoss-lab/slamming .

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 63 of its 81 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

63 of 81 references.

References · 63

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
38 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  2. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  4. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  5. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  6. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  7. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  10. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  11. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  12. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  13. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  14. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  15. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  16. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  17. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  18. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  19. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  20. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  21. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  22. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  23. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  24. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 101
  25. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 94
  26. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  27. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  28. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  29. 2021How to Train BERT with an Academic Budgetcited by 80
  30. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 54
  31. 2021The Zero Resource Speech Challenge 2021: Spoken language modellingcited by 47
  32. 2024Scaling Speech-Text Pre-training with Synthetic Interleaved Datacited by 47
  33. 2023Sparks of Large Audio Models: A Survey and Outlookcited by 41
  34. 2024Scaling Properties of Speech Language Modelscited by 32
  35. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 31
  36. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  37. 2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24
  38. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 4.

12 papers.

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 14
  3. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  4. 2025Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modelingcited by 7
  5. 2026Covo-Audio Technical Reportcited by 7
  6. 2025Advancing Speech Understanding in Speech-Aware Language Models with GRPOcited by 4
  7. 2025Scaling Spoken Language Models with Syllabic Speech Tokenizationcited by 2
  8. 2025Acoustic-based Gender Differentiation in Speech-aware Language Modelscited by 1
  9. 2025Cross-Lingual Interleaving for Speech Language Modelscited by 0
  10. 2026Interleaved Speech Language Models Latently Work In Textcited by 0
  11. 2026Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenizationcited by 0
  12. 2026Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models
  2. 2025Training and Inference Efficiency of Encoder-Decoder Speech Models
  3. 2026L20-Edu-135M: An Auditable Single-GPU Study of Data-Efficient Small Language Modeling
  4. 2025EfficientLLM: Efficiency in Large Language Models
  5. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuning
  6. 2025SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving
  7. 2026AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall ↗
  8. 2024Inference Performance Optimization for Large Language Models on CPUs