# Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Study

arXiv: https://arxiv.org/abs/2303.06740 · HTML: https://mlsanity.com/paper/2303.06740/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 12 Mar 2023.

## Builds on (most cited first, 23 of 23)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Self-supervised Learning with Random-projection Quantizer for Speech Recognition](https://mlsanity.com/paper/2202.01855/) (2022)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [On the Effect of Dropping Layers of Pre-trained Transformer Models](https://mlsanity.com/paper/2004.03844/) (2020)
- [A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition](https://mlsanity.com/paper/1807.10857/) (2018)
- [LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT](https://mlsanity.com/paper/2203.15610/) (2022)
- [How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communications](https://mlsanity.com/paper/2203.16822/) (2022)
- [SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2210.08634/) (2022)
- [FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning](https://mlsanity.com/paper/2207.00555/) (2022)
- [Evaluating the reliability of acoustic speech embeddings](https://mlsanity.com/paper/2007.13542/) (2020)
- [HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognition](https://mlsanity.com/paper/2204.06328/) (2022)
- [Pretext Tasks selection for multitask self-supervised speech representation learning](https://mlsanity.com/paper/2107.00594/) (2021)
- [Avoid Overthinking in Self-Supervised Models for Speech Recognition](https://mlsanity.com/paper/2211.08989/) (2022)
- [Efficient Speech Representation Learning with Low-Bit Quantization](https://mlsanity.com/paper/2301.00652/) (2022)
- [Automatic Data Augmentation Selection and Parametrization in Contrastive Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2204.04170/) (2022)
- [Once-for-All Sequence Compression for Self-Supervised Speech Models](https://mlsanity.com/paper/2211.02332/) (2022)

## Built on by (most cited first, 15 of 15)

- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Open-Source Conversational AI with SpeechBrain 1.0](https://mlsanity.com/paper/2407.00463/) (2024)
- [How Should We Extract Discrete Audio Tokens from Self-Supervised Models?](https://mlsanity.com/paper/2406.10735/) (2024)
- [Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Heads](https://mlsanity.com/paper/2308.14456/) (2023)
- [SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning](https://mlsanity.com/paper/2402.16830/) (2024)
- [Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularity](https://mlsanity.com/paper/2405.16579/) (2024)
- [Training dynamic models using early exits for automatic speech recognition on resource-constrained devices](https://mlsanity.com/paper/2309.09546/) (2023)
- [Big model only for hard audios: Sample dependent Whisper model selection for efficient inferences](https://mlsanity.com/paper/2309.12712/) (2023)
- [Input Conditioned Layer Dropping in Speech Foundation Models](https://mlsanity.com/paper/2507.07954/) (2025)
- [On the choice of the optimal temporal support for audio classification with Pre-trained embeddings](https://mlsanity.com/paper/2312.14005/) (2023)
- [An Effective Training Framework for Light-Weight Automatic Speech Recognition Models](https://mlsanity.com/paper/2505.16991/) (2025)
- [Splitformer: An improved early-exit architecture for automatic speech recognition on edge devices](https://mlsanity.com/paper/2506.18035/) (2025)
- [Distillation-based Layer Dropping (DLD): Effective End-to-end Framework for Dynamic Speech Networks](https://mlsanity.com/paper/2601.16117/) (2026)
- [Windowed SummaryMixing: An Efficient Fine-Tuning of Self-Supervised Learning Models for Low-resource Speech Recognition](https://mlsanity.com/paper/2602.09043/) (2026)
- [From Sinhala to Dhivehi: Cross-Lingual Transfer Learning for Low-Resource Speech Recognition](https://mlsanity.com/paper/2607.06289/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) has allowed substantial progress in Automatic Speech Recognition (ASR) performance in low-resource settings. In this context, it has been demonstrated that larger self-supervised feature extractors are crucial for achieving lower downstream ASR error rates. Thus, better performance might be sanctioned with longer inferences. This article explores different approaches that may be deployed during the fine-tuning to reduce the computations needed in the SSL encoder, leading to faster inferences. We adapt a number of existing techniques to common ASR settings and benchmark them, displaying performance drops and gains in inference times. Interestingly, we found that given enough downstream data, a simple downsampling of the input sequences outperforms the other methods with both low performance drops and high computational savings, reducing computations by 61.3% with an WER increase of only 0.81. Finally, we analyze the robustness of the comparison to changes in dataset conditions, revealing sensitivity to dataset size.