mlsanityData to 7 Oct 2026

Sustainable self-supervised learning for speech representations

arXiv 2406.07696 ↗First posted 11 Jun 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Sustainable artificial intelligence focuses on data, hardware, and algorithms to make machine learning models more environmentally responsible. In particular, machine learning models for speech representations are computationally expensive, generating environmental concerns because of their high energy consumption. Thus, we propose a sustainable self-supervised model to learn speech representation, combining optimizations in neural layers and training to reduce computing costs. The proposed model improves over a resource-efficient baseline, reducing both memory usage and computing cost estimations. It pretrains using a single GPU in less than a day. On top of that, it improves the error rate performance of the baseline in downstream task evaluations. When comparing it to large speech representation approaches, there is an order of magnitude reduction in memory usage, while computing cost reductions represent almost three orders of magnitude improvement.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 33 of its 41 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

33 of 41 references.

References · 33

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  10. 2017Mixed Precision Trainingcited by 1,135
  11. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
21 more
  1. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  3. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  4. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  5. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  6. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  7. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  8. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  9. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  10. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 34
  11. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
  12. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 29
  13. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  14. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
  15. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  16. 2022SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Trainingcited by 19
  17. 2023Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batchingcited by 12
  18. 2022On-demand compute reduction with stochastic wav2vec 2.0cited by 10
  19. 2023READ: Recurrent Adaptation of Large Transformerscited by 9
  20. 2022Match to Win: Analysing Sequences Lengths for Efficient Self-supervised Learning in Speech and Audiocited by 6
  21. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
  2. 2025Efficient Finetuning for Dimensional Speech Emotion Recognition in the Age of Transformerscited by 3
  3. 2025Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecscited by 3
  4. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognitioncited by 0