mlsanityData to 13 Oct 2026

HEMP: High-order Entropy Minimization for neural network comPression

arXiv 2107.05298 ↗First posted 12 Jul 2021cs.LG cs.AI cs.ITcited by 4

Not on the map. Why?

Abstract (arXiv)

We formulate the entropy of a quantized artificial neural network as a differentiable function that can be plugged as a regularization term into the cost function minimized by gradient descent. Our formulation scales efficiently beyond the first order and is agnostic of the quantization scheme. The network can then be trained to minimize the entropy of the quantized parameters, so that they can be optimally compressed via entropy coding. We experiment with our entropy formulation at quantizing and compressing well-known network architectures over multiple datasets. Our approach compares favorably over similar methods, enjoying the benefits of higher order entropy estimate, showing flexibility towards non-uniform quantization (we use Lloyd-max quantization), scalability towards any entropy order to be minimized and efficiency in terms of compression. We show that HEMP is able to work in synergy with other approaches aiming at pruning or quantizing the model itself, delivering significant benefits in terms of storage size compressibility without harming the model's performance.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 12 Jul 2021.

Builds on

23 of 34 references.

References · 23

  1. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,786
  2. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,315
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,257
  4. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,199
  5. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,058
  6. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  7. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 589
  8. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 574
  9. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 565
  10. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 439
  11. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  12. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 162
11 more
  1. 2017Towards Accurate Binary Convolutional Neural Networkcited by 149
  2. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  3. 2020APQ: Joint Search for Network Architecture, Pruning and Quantization Policycited by 98
  4. 2017WRPN: Wide Reduced-Precision Networkscited by 51
  5. 2019Scalable Model Compression by Entropy Penalized Reparameterizationcited by 31
  6. 2017Learning Discrete Weights Using the Local Reparameterization Trickcited by 29
  7. 2018Deep Neural Network Compression with Single and Multiple Level Quantizationcited by 27
  8. 2018Learning Sparse Neural Networks via Sensitivity-Driven Regularizationcited by 26
  9. 2020LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networkscited by 19
  10. 2018Adversarial Network Compressioncited by 9
  11. 2017On the role of synaptic stochasticity in training low-precision neural networkscited by 4

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 1; 2023: 1; 2024: 1.

4 papers.

  1. 2023Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?cited by 20
  2. 2021Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNscited by 8
  3. 2022Weight Fixing Networkscited by 1
  4. 2024Efficient Neural Compression with Inference-time Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding
  2. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
  3. 2025Optimizing Learned Image Compression on Scalar and Entropy-Constraint Quantization
  4. 2026Soft Quantization: Model Compression Via Weight Coupling
  5. 2024An Information-Theoretic Regularizer for Lossy Neural Image Compression
  6. 2025MEC-Quant: Maximum Entropy Coding for Extremely Low Bit Quantization-Aware Training
  7. 2025Theoretical Compression Bounds for Wide Multilayer Perceptrons
  8. 2021On the Acceleration of Deep Neural Network Inference using Quantized Compressed Sensing