Big model only for hard audios: Sample dependent Whisper model selection for efficient inferences
Not on the map. Why?
Abstract (arXiv)
Recent progress in Automatic Speech Recognition (ASR) has been coupled with a substantial increase in the model sizes, which may now contain billions of parameters, leading to slow inferences even with adapted hardware. In this context, several ASR models exist in various sizes, with different inference costs leading to different performance levels. Based on the observation that smaller models perform optimally on large parts of testing corpora, we propose to train a decision module, that would allow, given an audio sample, to use the smallest sufficient model leading to a good transcription. We apply our approach to two Whisper models with different sizes. By keeping the decision process computationally efficient, we build a decision module that allows substantial computational savings with reduced performance drops.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 28 of its 30 references that are in the map. First posted on arXiv 22 Sep 2023.
Builds on
28 of 30 references.
References · 28
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
- 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
- 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
- 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 120
16 more
- 2020Unsupervised pretraining transfers well across languagescited by 100
- 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
- 2022Probing phoneme, language and speaker information in unsupervised speech representationscited by 31
- 2023MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognitioncited by 31
- 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
- 2022Brouhaha: multi-task training for voice activity detection, speech-to-noise ratio, and C50 room acoustics estimationcited by 20
- 2022Hey ASR System! Why Aren't You More Inclusive? Automatic Speech Recognition Systems' Bias and Proposed Bias Mitigation Techniques. A Literature Reviewcited by 19
- 2023CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification Based on Common Voicecited by 19
- 2023Mixture-of-Expert Conformer for Streaming Multilingual ASRcited by 17
- 2023Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Headscited by 16
- 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
- 2022HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognitioncited by 13
- 2020Word Error Rate Estimation Without ASR Output: e-WER2cited by 8
- 2021WER-BERT: Automatic WER Estimation with BERT in a Balanced Ordinal Classification Paradigmcited by 5
- 2023Multilingual Word Error Rate Estimation: e-WER3cited by 5
- 2020Surprisal-Triggered Conditional Computation with Neural Networkscited by 4
Built on by
Citing papers per year: 2024: 2.
2 papers.