# Neural Network Compression Via Sparse Optimization

arXiv: https://arxiv.org/abs/2011.04868 · HTML: https://mlsanity.com/paper/2011.04868/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2020.

## Built on by (most cited first, 13 of 13)

- [CDFI: Compression-Driven Network Design for Frame Interpolation](https://mlsanity.com/paper/2103.10559/) (2021)
- [LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery](https://mlsanity.com/paper/2310.18356/) (2023)
- [OTOV2: Automatic, Generic, User-Friendly](https://mlsanity.com/paper/2303.06862/) (2023)
- [The Efficiency Spectrum of Large Language Models: An Algorithmic Survey](https://mlsanity.com/paper/2312.00678/) (2023)
- [ST-MFNet Mini: Knowledge Distillation-Driven Frame Interpolation](https://mlsanity.com/paper/2302.08455/) (2023)
- [Sparsity-guided Network Design for Frame Interpolation](https://mlsanity.com/paper/2209.04551/) (2022)
- [OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operators](https://mlsanity.com/paper/2312.09411/) (2023)
- [Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD](https://mlsanity.com/paper/2306.08125/) (2023)
- [HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning](https://mlsanity.com/paper/2409.09085/) (2024)
- [ONNXPruner: ONNX-Based General Model Pruning Adapter](https://mlsanity.com/paper/2404.08016/) (2024)
- [WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference](https://mlsanity.com/paper/2505.19427/) (2025)
- [FSCNN: A Fast Sparse Convolution Neural Network Inference System](https://mlsanity.com/paper/2212.08815/) (2022)
- [Review Neural Networks about Image Transformation Based on IGC Learning Framework with Annotated Information](https://mlsanity.com/paper/2206.10155/) (2022)

## Abstract (arXiv)

The compression of deep neural networks (DNNs) to reduce inference cost becomes increasingly important to meet realistic deployment requirements of various applications. There have been a significant amount of work regarding network compression, while most of them are heuristic rule-based or typically not friendly to be incorporated into varying scenarios. On the other hand, sparse optimization yielding sparse solutions naturally fits the compression requirement, but due to the limited study of sparse optimization in stochastic learning, its extension and application onto model compression is rarely well explored. In this work, we propose a model compression framework based on the recent progress on sparse stochastic optimization. Compared to existing model compression techniques, our method is effective and requires fewer extra engineering efforts to incorporate with varying applications, and has been numerically demonstrated on benchmark compression tasks. Particularly, we achieve up to 7.2 and 2.9 times FLOPs reduction with the same level of evaluation accuracy on VGG16 for CIFAR10 and ResNet50 for ImageNet compared to the baseline heavy models, respectively.