# Lossless Compression of Deep Neural Networks

arXiv: https://arxiv.org/abs/2001.00218 · HTML: https://mlsanity.com/paper/2001.00218/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jan 2020.

## Built on by (most cited first, 30 of 34)

- [Towards Lower Bounds on the Depth of ReLU Neural Networks](https://mlsanity.com/paper/2105.14835/) (2021)
- [The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks](https://mlsanity.com/paper/2203.04466/) (2022)
- [Partition-based formulations for mixed-integer optimization of trained ReLU neural networks](https://mlsanity.com/paper/2102.04373/) (2021)
- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [OMLT: Optimization & Machine Learning Toolkit](https://mlsanity.com/paper/2202.02414/) (2022)
- [Pruning's Effect on Generalization Through the Lens of Training and Regularization](https://mlsanity.com/paper/2210.13738/) (2022)
- [Parameter identifiability of a deep feedforward ReLU neural network](https://mlsanity.com/paper/2112.12982/) (2021)
- [Scaling Up Exact Neural Network Compression by ReLU Stability](https://mlsanity.com/paper/2102.07804/) (2021)
- [On the Expected Complexity of Maxout Networks](https://mlsanity.com/paper/2107.00379/) (2021)
- [Between steps: Intermediate relaxations between big-M and convex hull formulations](https://mlsanity.com/paper/2101.12708/) (2021)
- [Optimizing Objective Functions from Trained ReLU Neural Networks via Sampling](https://mlsanity.com/paper/2205.14189/) (2022)
- [Optimization Over Trained Neural Networks: Taking a Relaxing Walk](https://mlsanity.com/paper/2401.03451/) (2024)
- [Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithm](https://mlsanity.com/paper/2206.02976/) (2022)
- [Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regions](https://mlsanity.com/paper/2301.07966/) (2023)
- [Universal approximation and model compression for radial neural networks](https://mlsanity.com/paper/2107.02550/) (2021)
- [P-split formulations: A class of intermediate formulations between big-M and convex hull for disjunctive constraints](https://mlsanity.com/paper/2202.05198/) (2022)
- [Computational Tradeoffs of Optimization-Based Bound Tightening in ReLU Networks](https://mlsanity.com/paper/2312.16699/) (2023)
- [Deep Neural Networks pruning via the Structured Perspective Regularization](https://mlsanity.com/paper/2206.14056/) (2022)
- [Quiver neural networks](https://mlsanity.com/paper/2207.12773/) (2022)
- [Free Lunch in the Forest: Functionally-Identical Pruning of Boosted Tree Ensembles](https://mlsanity.com/paper/2408.16167/) (2024)
- [On the Impact of Stable Ranks in Deep Nets](https://mlsanity.com/paper/2110.02333/) (2021)
- [Training Thinner and Deeper Neural Networks: Jumpstart Regularization](https://mlsanity.com/paper/2201.12795/) (2022)
- [Test-Time Steering for Lossless Text Compression via Weighted Product of Experts](https://mlsanity.com/paper/2511.10660/) (2025)
- [Support Vector Machines with the Hard-Margin Loss: Optimal Training via Combinatorial Benders' Cuts](https://mlsanity.com/paper/2207.07690/) (2022)
- [The Effect of Data Dimensionality on Neural Network Prunability](https://mlsanity.com/paper/2212.00291/) (2022)
- [Model-based feature selection for neural networks: A mixed-integer programming approach](https://mlsanity.com/paper/2302.10344/) (2023)
- [Optimizing over an ensemble of neural networks](https://mlsanity.com/paper/2112.07007/) (2021)
- [Multi-Objective Linear Ensembles for Robust and Sparse Training of Few-Bit Neural Networks](https://mlsanity.com/paper/2212.03659/) (2022)
- [Learning Prescriptive ReLU Networks](https://mlsanity.com/paper/2306.00651/) (2023)
- [Towards Global Neural Network Abstractions with Locally-Exact Reconstruction](https://mlsanity.com/paper/2210.12054/) (2022)

## Abstract (arXiv)

Deep neural networks have been successful in many predictive modeling tasks, such as image and language recognition, where large neural networks are often used to obtain good accuracy. Consequently, it is challenging to deploy these networks under limited computational resources, such as in mobile devices. In this work, we introduce an algorithm that removes units and layers of a neural network while not changing the output that is produced, which thus implies a lossless compression. This algorithm, which we denote as LEO (Lossless Expressiveness Optimization), relies on Mixed-Integer Linear Programming (MILP) to identify Rectified Linear Units (ReLUs) with linear behavior over the input domain. By using L1 regularization to induce such behavior, we can benefit from training over a larger architecture than we would later use in the environment where the trained neural network is deployed.