mlsanityData to 7 Oct 2026

Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Study

arXiv 2303.06740 ↗First posted 12 Mar 2023eess.AS cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has allowed substantial progress in Automatic Speech Recognition (ASR) performance in low-resource settings. In this context, it has been demonstrated that larger self-supervised feature extractors are crucial for achieving lower downstream ASR error rates. Thus, better performance might be sanctioned with longer inferences. This article explores different approaches that may be deployed during the fine-tuning to reduce the computations needed in the SSL encoder, leading to faster inferences. We adapt a number of existing techniques to common ASR settings and benchmark them, displaying performance drops and gains in inference times. Interestingly, we found that given enough downstream data, a simple downsampling of the input sequences outperforms the other methods with both low performance drops and high computational savings, reducing computations by 61.3% with an WER increase of only 0.81. Finally, we analyze the robustness of the comparison to changes in dataset conditions, revealing sensitivity to dataset size.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 12 Mar 2023.

Builds on

23 of 29 references.

References · 23

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  6. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  7. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  8. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  9. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  10. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  11. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  12. 2018A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognitioncited by 66
11 more
  1. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  2. 2022How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communicationscited by 30
  3. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  4. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
  5. 2020Evaluating the reliability of acoustic speech embeddingscited by 21
  6. 2022HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognitioncited by 13
  7. 2021Pretext Tasks selection for multitask self-supervised speech representation learningcited by 7
  8. 2022Avoid Overthinking in Self-Supervised Models for Speech Recognitioncited by 5
  9. 2022Automatic Data Augmentation Selection and Parametrization in Contrastive Self-Supervised Speech Representation Learningcited by 4
  10. 2022Efficient Speech Representation Learning with Low-Bit Quantizationcited by 4
  11. 2022Once-for-All Sequence Compression for Self-Supervised Speech Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 4; 2025: 3; 2026: 3.

15 papers.

Sort:

By area:

  1. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  2. 2024Open-Source Conversational AI with SpeechBrain 1.0cited by 55
  3. 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24
  4. 2023Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Headscited by 16
  5. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learningcited by 4
  6. 2024Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularitycited by 4
  7. 2023Training dynamic models using early exits for automatic speech recognition on resource-constrained devicescited by 3
  8. 2023Big model only for hard audios: Sample dependent Whisper model selection for efficient inferencescited by 2
  9. 2025Input Conditioned Layer Dropping in Speech Foundation Modelscited by 2
  10. 2023On the choice of the optimal temporal support for audio classification with Pre-trained embeddingscited by 1
  11. 2025An Effective Training Framework for Light-Weight Automatic Speech Recognition Modelscited by 1
  12. 2025Splitformer: An improved early-exit architecture for automatic speech recognition on edge devicescited by 1
3 more
  1. 2026Distillation-based Layer Dropping (DLD): Effective End-to-end Framework for Dynamic Speech Networkscited by 0
  2. 2026Windowed SummaryMixing: An Efficient Fine-Tuning of Self-Supervised Learning Models for Low-resource Speech Recognitioncited by 0
  3. 2026From Sinhala to Dhivehi: Cross-Lingual Transfer Learning for Low-Resource Speech Recognitioncited by 0