# Sparsification via Compressed Sensing for Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2102.04932 · HTML: https://mlsanity.com/paper/2102.04932/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Exploring Sparsity in Recurrent Neural Networks](https://mlsanity.com/paper/1704.05119/) (2017)
- [Streaming End-to-End Bilingual ASR Systems with Joint Language Identification](https://mlsanity.com/paper/2007.03900/) (2020)
- [Image Compression Based on Compressive Sensing: End-to-End Comparison with JPEG](https://mlsanity.com/paper/1706.01000/) (2017)

## Built on by (most cited first, 8 of 8)

- [Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models](https://mlsanity.com/paper/2110.03334/) (2021)
- [CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition](https://mlsanity.com/paper/2106.07734/) (2021)
- [Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognition](https://mlsanity.com/paper/2206.15408/) (2022)
- [Representative Subset Selection for Efficient Fine-Tuning in Self-Supervised Speech Recognition](https://mlsanity.com/paper/2203.09829/) (2022)
- [An Effective Training Framework for Light-Weight Automatic Speech Recognition Models](https://mlsanity.com/paper/2505.16991/) (2025)
- [Accelerator-Aware Training for Transducer-Based Speech Recognition](https://mlsanity.com/paper/2305.07778/) (2023)
- [Neural Language Model Pruning for Automatic Speech Recognition](https://mlsanity.com/paper/2310.03424/) (2023)
- [Gibbs randomness-compression proposition](https://mlsanity.com/paper/2505.23869/) (2025)

## Abstract (arXiv)

In order to achieve high accuracy for machine learning (ML) applications, it is essential to employ models with a large number of parameters. Certain applications, such as Automatic Speech Recognition (ASR), however, require real-time interactions with users, hence compelling the model to have as low latency as possible. Deploying large scale ML applications thus necessitates model quantization and compression, especially when running ML models on resource constrained devices. For example, by forcing some of the model weight values into zero, it is possible to apply zero-weight compression, which reduces both the model size and model reading time from the memory. In the literature, such methods are referred to as sparse pruning. The fundamental questions are when and which weights should be forced to zero, i.e. be pruned. In this work, we propose a compressed sensing based pruning (CSP) approach to effectively address those questions. By reformulating sparse pruning as a sparsity inducing and compression-error reduction dual problem, we introduce the classic compressed sensing process into the ML model training process. Using ASR task as an example, we show that CSP consistently outperforms existing approaches in the literature.