# Fast Convex Pruning of Deep Neural Networks

arXiv: https://arxiv.org/abs/1806.06457 · HTML: https://mlsanity.com/paper/1806.06457/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2018.

## Built on by (most cited first, 8 of 8)

- [Gradient-Free Structured Pruning with Unlabeled Data](https://mlsanity.com/paper/2303.04185/) (2023)
- [RIGID: Robust Linear Regression with Missing Data](https://mlsanity.com/paper/2205.13635/) (2022)
- [State-driven Implicit Modeling for Sparsity and Robustness in Neural Networks](https://mlsanity.com/paper/2209.09389/) (2022)
- [A Proximal Algorithm for Network Slimming](https://mlsanity.com/paper/2307.00684/) (2023)
- [Feather: An Elegant Solution to Effective DNN Sparsification](https://mlsanity.com/paper/2310.02448/) (2023)
- [Tensor-based framework for training flexible neural networks](https://mlsanity.com/paper/2106.13542/) (2021)
- [Holistic Deep Learning](https://mlsanity.com/paper/2110.15829/) (2021)
- [Weight Matrix Dimensionality Reduction in Deep Learning via Kronecker Multi-layer Architectures](https://mlsanity.com/paper/2204.04273/) (2022)

## Abstract (arXiv)

We develop a fast, tractable technique called Net-Trim for simplifying a trained neural network. The method is a convex post-processing module, which prunes (sparsifies) a trained network layer by layer, while preserving the internal responses. We present a comprehensive analysis of Net-Trim from both the algorithmic and sample complexity standpoints, centered on a fast, scalable convex optimization program. Our analysis includes consistency results between the initial and retrained models before and after Net-Trim application and guarantees on the number of training samples needed to discover a network that can be expressed using a certain number of nonzero terms. Specifically, if there is a set of weights that uses at most s terms that can re-create the layer outputs from the layer inputs, we can find these weights from O(slog N/s) samples, where N is the input size. These theoretical results are similar to those for sparse regression using the Lasso, and our analysis uses some of the same recently-developed tools (namely recent results on the concentration of measure and convex analysis). Finally, we propose an algorithmic framework based on the alternating direction method of multipliers (ADMM), which allows a fast and simple implementation of Net-Trim for network pruning and compression.