mlsanityData to 14 Oct 2026

HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognition

arXiv 2204.06328 ↗First posted 13 Apr 2022cs.CL cs.SD eess.AScited by 13

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Pre-training with self-supervised models, such as Hidden-unit BERT (HuBERT) and wav2vec 2.0, has brought significant improvements in automatic speech recognition (ASR). However, these models usually require an expensive computational cost to achieve outstanding performance, slowing down the inference speed. To improve the model efficiency, we introduce an early exit scheme for ASR, namely HuBERT-EE, that allows the model to stop the inference dynamically. In HuBERT-EE, multiple early exit branches are added at the intermediate layers. When the intermediate prediction of the early exit branch is confident, the model stops the inference, and the corresponding result can be returned early. We investigate the proper early exiting criterion and fine-tuning strategy to effectively perform early exiting. Experimental results on the LibriSpeech show that HuBERT-EE can accelerate the inference of the HuBERT while simultaneously balancing the trade-off between the performance and the latency.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 33 references that are in the map. First posted on arXiv 13 Apr 2022.

Builds on

25 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Elbert: Fast Albert with Confidence-Window Based Early Exit1 Jul

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  11. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  12. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
12 more
  1. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  2. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  3. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  4. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 215
  5. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  6. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 126
  7. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 96
  8. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  9. 2020Don't shoot butterfly with rifles: Multi-channel Continuous Speech Separation with Early Exit Transformercited by 16
  10. 2021Self-Supervised Learning for speech recognition with Intermediate layer supervisioncited by 15
  11. 2021Layer Pruning on Demand with Intermediate CTCcited by 13
  12. 2021Learning to Inference with Early Exit in the Progressive Speech Enhancementcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 2; 2025: 4; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 37
  2. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
  3. 2025Token Pruning in Audio Transformers: Optimizing Performance and Decoding Patch Importancecited by 12
  4. 2023Subnetwork-to-go: Elastic Neural Network with Dynamic Training and Customizable Inferencecited by 9
  5. 2022Avoid Overthinking in Self-Supervised Models for Speech Recognitioncited by 5
  6. 2022Understanding the Robustness of Multi-Exit Models under Common Corruptionscited by 4
  7. 2024DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Modelscited by 4
  8. 2023Training dynamic models using early exits for automatic speech recognition on resource-constrained devicescited by 3
  9. 2023Big model only for hard audios: Sample dependent Whisper model selection for efficient inferencescited by 2
  10. 2025Input Conditioned Layer Dropping in Speech Foundation Modelscited by 2
  11. 2025Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Samplingcited by 0
  12. 2025Spiralformer: Low Latency Encoder for Streaming Speech Recognition with Circular Layer Skipping and Early Exitingcited by 0
1 more
  1. 2026SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference
  2. 2025DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective
  3. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition
  4. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  5. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  6. 2021RomeBERT: Robust Training of Multi-Exit BERT
  7. 2026The Diminishing Returns of Early-Exit Decoding in Modern LLMs
  8. 2024UME: Upcycling Mixture-of-Experts for Scalable and Efficient Automatic Speech Recognition