mlsanityData to 7 Oct 2026

Layer Pruning on Demand with Intermediate CTC

arXiv 2106.09216 ↗First posted 17 Jun 2021eess.AS cs.CL cs.SDcited by 13

Not on the map. Why?

Abstract (arXiv)

Deploying an end-to-end automatic speech recognition (ASR) model on mobile/embedded devices is a challenging task, since the device computational power and energy consumption requirements are dynamically changed in practice. To overcome the issue, we present a training and pruning method for ASR based on the connectionist temporal classification (CTC) which allows reduction of model depth at run-time without any extra fine-tuning. To achieve the goal, we adopt two regularization methods, intermediate CTC and stochastic depth, to train a model whose performance does not degrade much after pruning. We present an in-depth analysis of layer behaviors using singular vector canonical correlation analysis (SVCCA), and efficient strategies for finding layers which are safe to prune. Using the proposed method, we show that a Transformer-CTC model can be pruned in various depth on demand, improving real-time factor from 0.005 to 0.002 on GPU, while each pruned sub-model maintains the accuracy of individually trained model of the same depth.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

29 of 40 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2016Deep Networks with Stochastic Depthcited by 973
  9. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  10. 2015Listen, Attend and Spellcited by 542
  11. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  12. 2015Attention-Based Models for Speech Recognitioncited by 400
17 more
  1. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  2. 2018Slimmable Neural Networkscited by 301
  3. 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
  4. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  5. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 95
  6. 2019QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutionscited by 93
  7. 2019AI Benchmark: All About Deep Learning on Smartphones in 2019cited by 76
  8. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 75
  9. 2020Imputer: Sequence Modelling via Imputation and Dynamic Programmingcited by 65
  10. 2019Very Deep Self-Attention Networks for End-to-End Speech Recognitioncited by 46
  11. 2020Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignmentcited by 37
  12. 2019On-Device Neural Net Inference with Mobile GPUscited by 32
  13. 2020Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognitioncited by 21
  14. 2020Joint Masked CPC and CTC Training for ASRcited by 20
  15. 2020Scaling Up Online Speech Recognition Using ConvNetscited by 16
  16. 2017Building competitive direct acoustics-to-word models for English conversational speech recognitioncited by 10
  17. 2020Any-Width Networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 2; 2024: 3; 2025: 4; 2026: 1.

13 papers.

Sort:

By area:

  1. 2021Hierarchical Conditional End-to-End ASR with CTC and Multi-Granular Subword Unitscited by 13
  2. 2022HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognitioncited by 13
  3. 2022BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Modelcited by 11
  4. 2023I3D: Transformer architectures with input-dependent dynamic depth for speech recognitioncited by 10
  5. 2024Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognitioncited by 7
  6. 2023A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognitioncited by 5
  7. 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
  8. 2024One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Modelcited by 4
  9. 2024Guiding Frame-Level CTC Alignments Using Self-knowledge Distillationcited by 1
  10. 2025Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compressioncited by 1
  11. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
  12. 2025Spiralformer: Low Latency Encoder for Streaming Speech Recognition with Circular Layer Skipping and Early Exitingcited by 0
1 more
  1. 2026Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clusteringcited by 0