mlsanityData to 7 Oct 2026

Accurate and Structured Pruning for Efficient Automatic Speech Recognition

arXiv 2305.19549 ↗First posted 31 May 2023cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Automatic Speech Recognition (ASR) has seen remarkable advancements with deep neural networks, such as Transformer and Conformer. However, these models typically have large model sizes and high inference costs, posing a challenge to deploy on resource-limited devices. In this paper, we propose a novel compression strategy that leverages structured pruning and knowledge distillation to reduce the model size and inference cost of the Conformer model while preserving high recognition performance. Our approach utilizes a set of binary masks to indicate whether to retain or prune each Conformer module, and employs L0 regularization to learn the optimal mask values. To further enhance pruning performance, we use a layerwise distillation strategy to transfer knowledge from unpruned to pruned models. Our method outperforms all pruning baselines on the widely used LibriSpeech benchmark, achieving a 50% reduction in model size and a 28% reduction in inference cost with minimal performance loss.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

24 of 31 references.

References · 24

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  9. 2019Are Sixteen Heads Really Better than One?cited by 803
  10. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  11. 2019Importance Estimation for Neural Network Pruningcited by 531
  12. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
12 more
  1. 2019Structured Pruning of Large Language Modelscited by 201
  2. 2021Block Pruning For Faster Transformerscited by 176
  3. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  4. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  5. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  6. 2021PQK: Model Compression via Pruning, Quantization, and Knowledge Distillationcited by 19
  7. 2022PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representationscited by 10
  8. 2022On Compressing Sequences for Self-Supervised Speech Modelscited by 7
  9. 2022Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognitioncited by 6
  10. 2022Improving generalizability of distilled self-supervised speech processing models under distorted settingscited by 5
  11. 2022SwiftPruner: Reinforced Evolutionary Pruning for Efficient Ad Relevancecited by 4
  12. 2022Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 3; 2026: 3.

10 papers.

  1. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  2. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 7
  3. 2024Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognitioncited by 7
  4. 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
  5. 2024One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Modelcited by 4
  6. 2025Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compressioncited by 1
  7. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
  8. 2026Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognitioncited by 0
  9. 2026Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clusteringcited by 0
  10. 2026VAD to the Bone: Ultra-Tiny Speech Activity Detection for Edge Deploymentcited by 0