mlsanityData to 7 Oct 2026

Neural Language Model Pruning for Automatic Speech Recognition

arXiv 2310.03424 ↗First posted 5 Oct 2023cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We study model pruning methods applied to Transformer-based neural network language models for automatic speech recognition. We explore three aspects of the pruning frame work, namely criterion, method and scheduler, analyzing their contribution in terms of accuracy and inference speed. To the best of our knowledge, such in-depth analyses on large-scale recognition systems has not been reported in the literature. In addition, we propose a variant of low-rank approximation suitable for incrementally compressing models, and delivering multiple models with varied target sizes. Among other results, we show that a) data-driven pruning outperforms magnitude-driven in several scenarios; b) incremental pruning achieves higher accuracy compared to one-shot pruning, especially when targeting smaller sizes; and c) low-rank approximation presents the best trade-off between size reduction and inference speed-up for moderate compression.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

39 of 50 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2020Knowledge Distillation: A Surveycited by 1,578
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  11. 2018Self-Attention with Relative Position Representationscited by 1,112
  12. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
27 more
  1. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  2. 2019Importance Estimation for Neural Network Pruningcited by 531
  3. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  4. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  5. 2016Trained Ternary Quantizationcited by 203
  6. 2019Structured Pruning of Large Language Modelscited by 201
  7. 2022Language model compression with weighted low-rank factorizationcited by 192
  8. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  9. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  10. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  11. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  12. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
  13. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
  14. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  15. 2022Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Modelscited by 52
  16. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  17. 2018Filter Distillation for Network Compressioncited by 17
  18. 2018Structured Pruning for Efficient ConvNets via Incremental Regularizationcited by 15
  19. 2022No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Modelscited by 15
  20. 2022Variable Attention Masking for Configurable Transformer Transducer Speech Recognitioncited by 14
  21. 2020Layer-Wise Data-Free CNN Compressioncited by 13
  22. 2021GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsitycited by 10
  23. 2015Auto-Sizing Neural Networks: With Applications to n-gram Language Modelscited by 9
  24. 2021Error-driven Pruning of Language Models for Virtual Assistantscited by 8
  25. 2021Sparsification via Compressed Sensing for Automatic Speech Recognitioncited by 7
  26. 2022Space-Efficient Representation of Entity-centric Query Language Modelscited by 7
  27. 2022The Importance of Being Parameters: An Intra-Distillation Method for Serious Gainscited by 6

Built on by

Not cited yet.