mlsanityData to 7 Oct 2026

Match to Win: Analysing Sequences Lengths for Efficient Self-supervised Learning in Speech and Audio

arXiv 2209.15575 ↗First posted 30 Sep 2022cs.SD cs.LG eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has proven vital in speech and audio-related applications. The paradigm trains a general model on unlabeled data that can later be used to solve specific downstream tasks. This type of model is costly to train as it requires manipulating long input sequences that can only be handled by powerful centralised servers. Surprisingly, despite many attempts to increase training efficiency through model compression, the effects of truncating input sequence lengths to reduce computation have not been studied. In this paper, we provide the first empirical study of SSL pre-training for different specified sequence lengths and link this to various downstream tasks. We find that training on short sequences can dramatically reduce resource costs while retaining a satisfactory performance for all tasks. This simple one-line change would promote the migration of SSL training from data centres to user-end edge devices for more realistic and personalised applications.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 17 of its 24 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

17 of 24 references.

References · 17

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2017Mixed Precision Trainingcited by 1,135
  6. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  10. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  11. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  12. 2020Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representationcited by 58
5 more
  1. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  2. 2016Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognitioncited by 33
  3. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
  4. 2021Timers and Such: A Practical Benchmark for Spoken Language Understanding with Numberscited by 11
  5. 2022Federated Self-supervised Speech Representations: Are We There Yet?cited by 10

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 3.

6 papers.

  1. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17
  2. 2024Training Machine Learning models at the Edge: A Surveycited by 11
  3. 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
  4. 2024Sustainable self-supervised learning for speech representationscited by 4
  5. 2022Once-for-All Sequence Compression for Self-Supervised Speech Modelscited by 2
  6. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1