mlsanityData to 7 Oct 2026

Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognition

arXiv 2407.18930 ↗First posted 10 Jul 2024eess.AS cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Varying-size models are often required to deploy ASR systems under different hardware and/or application constraints such as memory and latency. To avoid redundant training and optimization efforts for individual models of different sizes, we present the dynamic encoder size approach, which jointly trains multiple performant models within one supernet from scratch. These subnets of various sizes are layer-wise pruned from the supernet, and thus, enjoy full parameter sharing. By combining score-based pruning with supernet training, we propose two novel methods, Simple-Top-k and Iterative-Zero-Out, to automatically select the best-performing subnets in a data-driven manner, avoiding resource-intensive search efforts. Our experiments using CTC on both Librispeech and TED-LIUM-v2 corpora show that our methods can achieve on-par performance as individually trained models of each size category. Also, our approach consistently brings small performance improvements for the full-size supernet.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 33 of its 37 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

33 of 37 references.

References · 33

  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2016Deep Networks with Stochastic Depthcited by 973
  6. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  7. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  8. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  9. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  10. 2018Slimmable Neural Networkscited by 301
  11. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
  12. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
21 more
  1. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  2. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 95
  3. 2019Reparameterizable Subset Sampling via Continuous Relaxationscited by 69
  4. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 68
  5. 2020Cascaded encoders for unifying streaming and non-streaming ASRcited by 51
  6. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  7. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  8. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 29
  9. 2019Deja-vu: Double Feature Presentation and Iterated Loss in Deep Transformer Networkscited by 24
  10. 2020Phoneme Based Neural Transducer for Large Vocabulary Speech Recognitioncited by 20
  11. 2020The RWTH ASR System for TED-LIUM Release 2: Improving Hybrid HMM with SpecAugmentcited by 19
  12. 2022Efficient Training of Neural Transducer for Speech Recognitioncited by 14
  13. 2021Layer Pruning on Demand with Intermediate CTCcited by 13
  14. 2023Task-Agnostic Structured Pruning of Speech Representation Modelscited by 13
  15. 2021Omni-sparsity DNN: Fast Sparsity Optimization for On-Device Streaming E2E ASR via Supernetcited by 12
  16. 2022A Unified Cascaded Encoder ASR Model for Dynamic Model Sizescited by 11
  17. 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognitioncited by 10
  18. 2021Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latencycited by 9
  19. 2021Collaborative Training of Acoustic Encoders for Speech Recognitioncited by 7
  20. 2022Enhancing and Adversarial: Improve ASR with Speaker Labelscited by 6
  21. 2023TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Label-Context-Dependent Internal Language Model Estimation for CTCcited by 2
  2. 2025Input Conditioned Layer Dropping in Speech Foundation Modelscited by 2
  3. 2025Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compressioncited by 1
  4. 2025Dynamic Acoustic Model Architecture Optimization in Training for ASRcited by 1
  5. 2025PARCO: Phoneme-Augmented Robust Contextual ASR via Contrastive Entity Disambiguationcited by 0
  6. 2026Distillation-based Layer Dropping (DLD): Effective End-to-end Framework for Dynamic Speech Networkscited by 0
  7. 2026X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillationcited by 0