mlsanityData to 14 Oct 2026

Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget

arXiv 2409.16295 ↗First posted 9 Sep 2024eess.AS cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Despite their impressive success, training foundation models remains computationally costly. This paper investigates how to efficiently train speech foundation models with self-supervised learning (SSL) under a limited compute budget. We examine critical factors in SSL that impact the budget, including model architecture, model size, and data size. Our goal is to make analytical steps toward understanding the training dynamics of speech foundation models. We benchmark SSL objectives in an entirely comparable setting and find that other factors contribute more significantly to the success of SSL. Our results show that slimmer model architectures outperform common small architectures under the same compute and parameter budget. We demonstrate that the size of the pre-training data remains crucial, even with data augmentation during SSL training, as performance suffers when iterating over limited data. Finally, we identify a trade-off between model size and data size, highlighting an optimal model size for a given compute budget.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 9 Sep 2024.

Builds on

32 of 39 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  8. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  9. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  10. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  11. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  12. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
20 more
  1. 2020Unsupervised pretraining transfers well across languagescited by 101
  2. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  3. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  4. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  5. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  6. 2020Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representationcited by 59
  7. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  8. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  9. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 40
  10. 2020Learning Robust and Multilingual Speech Representationscited by 39
  11. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  12. 2020Similarity Analysis of Self-Supervised Speech Representationscited by 26
  13. 2021Don't speak too fast: The impact of data bias on self-supervised speech modelscited by 20
  14. 2023Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspacescited by 20
  15. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 16
  16. 2024On the social bias of speech self-supervised modelscited by 16
  17. 2020Understanding Self-Attention of Self-Supervised Audio Transformerscited by 15
  18. 2022On-demand compute reduction with stochastic wav2vec 2.0cited by 10
  19. 2022Match to Win: Analysing Sequences Lengths for Efficient Self-supervised Learning in Speech and Audiocited by 6
  20. 2024Sustainable self-supervised learning for speech representationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 12
  2. 2025WaveSP-Net: Learnable Wavelet-Domain Sparse Prompt Tuning for Speech Deepfake Detectioncited by 5
  3. 2025Mixtraining: A Better Trade-Off Between Compute and Performancecited by 2
  4. 2025A correlation-permutation approach for speech-music encoders model mergingcited by 2
  5. 2026Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokenscited by 0