# Scalable Model Compression by Entropy Penalized Reparameterization

arXiv: https://arxiv.org/abs/1906.06624 · HTML: https://mlsanity.com/paper/1906.06624/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2019.

## Built on by (most cited first, 30 of 31)

- [EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS](https://mlsanity.com/paper/2312.04564/) (2023)
- [NIRVANA: Neural Implicit Representations of Videos with Adaptive Networks and Autoregressive Patch-wise Modeling](https://mlsanity.com/paper/2212.14593/) (2022)
- [The Fundamental Price of Secure Aggregation in Differentially Private Federated Learning](https://mlsanity.com/paper/2203.03761/) (2022)
- [SHACIRA: Scalable HAsh-grid Compression for Implicit Neural Representations](https://mlsanity.com/paper/2309.15848/) (2023)
- [Boosting Neural Representations for Videos with a Conditional Decoder](https://mlsanity.com/paper/2402.18152/) (2024)
- [An Information-Theoretic Justification for Model Pruning](https://mlsanity.com/paper/2102.08329/) (2021)
- [3D Scene Compression through Entropy Penalized Neural Representation Functions](https://mlsanity.com/paper/2104.12456/) (2021)
- [LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks](https://mlsanity.com/paper/2111.08988/) (2021)
- [NeRFCodec: Neural Feature Compression Meets Neural Radiance Fields for Memory-Efficient Scene Representation](https://mlsanity.com/paper/2404.02185/) (2024)
- [ZipNN: Lossless Compression for AI Models](https://mlsanity.com/paper/2411.05239/) (2024)
- [RECOMBINER: Robust and Enhanced Compression with Bayesian Implicit Neural Representations](https://mlsanity.com/paper/2309.17182/) (2023)
- [Neural Network Compression for Noisy Storage Devices](https://mlsanity.com/paper/2102.07725/) (2021)
- [ECRF: Entropy-Constrained Neural Radiance Fields Compression with Frequency Domain Optimization](https://mlsanity.com/paper/2311.14208/) (2023)
- [LilNetX: Lightweight Networks with EXtreme Model Compression and Structured Sparsification](https://mlsanity.com/paper/2204.02965/) (2022)
- [Lossless and Near-Lossless Compression for Foundation Models](https://mlsanity.com/paper/2404.15198/) (2024)
- [HEMP: High-order Entropy Minimization for neural network comPression](https://mlsanity.com/paper/2107.05298/) (2021)
- [Lossy and Lossless (L2) Post-training Model Size Compression](https://mlsanity.com/paper/2308.04269/) (2023)
- [Neural NeRF Compression](https://mlsanity.com/paper/2406.08943/) (2024)
- [Binarized Neural Networks Converge Toward Algorithmic Simplicity: Empirical Support for the Learning-as-Compression Hypothesis](https://mlsanity.com/paper/2505.20646/) (2025)
- [Deep Learning Inference Frameworks Benchmark](https://mlsanity.com/paper/2210.04323/) (2022)
- [Federated Automatic Differentiation](https://mlsanity.com/paper/2301.07806/) (2023)
- [From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks](https://mlsanity.com/paper/2405.06038/) (2024)
- [A Deep Neural Networks ensemble workflow from hyperparameter search to inference leveraging GPU clusters](https://mlsanity.com/paper/2208.14046/) (2022)
- [Rotation Invariant Quantization for Model Compression](https://mlsanity.com/paper/2303.03106/) (2023)
- [Bandwidth-efficient Inference for Neural Image Compression](https://mlsanity.com/paper/2309.02855/) (2023)
- [Neural Embedding Compression For Efficient Multi-Task Earth Observation Modelling](https://mlsanity.com/paper/2403.17886/) (2024)
- [DietCNN: Multiplication-free Inference for Quantized CNNs](https://mlsanity.com/paper/2305.05274/) (2023)
- [Federated Hyperdimensional Computing](https://mlsanity.com/paper/2312.15966/) (2023)
- [Efficient Neural Compression with Inference-time Decoding](https://mlsanity.com/paper/2406.06237/) (2024)
- [MOGNET: A Mux-residual quantized Network leveraging Online-Generated weights](https://mlsanity.com/paper/2501.09531/) (2025)

## Abstract (arXiv)

We describe a simple and general neural network weight compression approach, in which the network parameters (weights and biases) are represented in a "latent" space, amounting to a reparameterization. This space is equipped with a learned probability model, which is used to impose an entropy penalty on the parameter representation during training, and to compress the representation using a simple arithmetic coder after training. Classification accuracy and model compressibility is maximized jointly, with the bitrate--accuracy trade-off specified by a hyperparameter. We evaluate the method on the MNIST, CIFAR-10 and ImageNet classification benchmarks using six distinct model architectures. Our results show that state-of-the-art model compression can be achieved in a scalable and general way without requiring complex procedures such as multi-stage training.