# Self-Compressing Neural Networks

arXiv: https://arxiv.org/abs/2301.13142 · HTML: https://mlsanity.com/paper/2301.13142/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 15 references that are in the map.
- First posted on arXiv 30 Jan 2023.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)
- [Differentiable Model Compression via Pseudo Quantization Noise](https://mlsanity.com/paper/2104.09987/) (2021)
- [LUTNet: Learning FPGA Configurations for Highly Efficient Neural Network Inference](https://mlsanity.com/paper/1910.12625/) (2019)
- [Partitioning sparse deep neural networks for scalable training and inference](https://mlsanity.com/paper/2104.11805/) (2021)

## Built on by (most cited first, 2 of 2)

- [Optimizing Deep Neural Networks using Safety-Guided Self Compression](https://mlsanity.com/paper/2505.00350/) (2025)
- [Weight Pair Encoding: Inducing a Smaller Grammar in Neural Network Weights](https://mlsanity.com/paper/2609.31564/) (2026)

## Abstract (arXiv)

This work focuses on reducing neural network size, which is a major driver of neural network execution time, power consumption, bandwidth, and memory footprint. A key challenge is to reduce size in a manner that can be exploited readily for efficient training and inference without the need for specialized hardware. We propose Self-Compression: a simple, general method that simultaneously achieves two goals: (1) removing redundant weights, and (2) reducing the number of bits required to represent the remaining weights. This is achieved using a generalized loss function to minimize overall network size. In our experiments we demonstrate floating point accuracy with as few as 3% of the bits and 18% of the weights remaining in the network.