mlsanityData to 7 Oct 2026

Rotation Invariant Quantization for Model Compression

arXiv 2303.03106 ↗First posted 3 Mar 2023cs.LG cs.AI cs.ITcited by 1

Not on the map. Why?

Abstract (arXiv)

Post-training Neural Network (NN) model compression is an attractive approach for deploying large, memory-consuming models on devices with limited memory resources. In this study, we investigate the rate-distortion tradeoff for NN model compression. First, we suggest a Rotation-Invariant Quantization (RIQ) technique that utilizes a single parameter to quantize the entire NN model, yielding a different rate at each layer, i.e., mixed-precision quantization. Then, we prove that our rotation-invariant approach is optimal in terms of compression. We rigorously evaluate RIQ and demonstrate its capabilities on various models and tasks. For example, RIQ facilitates × 19.4 and × 52.9 compression ratios on pre-trained VGG dense and pruned models, respectively, with <0.4% accuracy degradation. Code is available in hrefhttps://github.com/ehaleva/RIQgithub.com/ehaleva/RIQ.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 65 references that are in the map. First posted on arXiv 3 Mar 2023.

Builds on

35 of 65 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  10. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  11. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  12. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
23 more
  1. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  2. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
  3. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  4. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 180
  5. 2020Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimizationcited by 176
  6. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  7. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  8. 2021High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspacescited by 139
  9. 2021Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvementcited by 115
  10. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  11. 2021Multi-Objective Bayesian Optimization over High-Dimensional Search Spacescited by 87
  12. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  13. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 44
  14. 2022Post-training Quantization for Neural Networks with Provable Guaranteescited by 32
  15. 2019Scalable Model Compression by Entropy Penalized Reparameterizationcited by 31
  16. 2021An Information-Theoretic Justification for Model Pruningcited by 23
  17. 2020Online Ensemble Model Compression using Knowledge Distillationcited by 16
  18. 2020Knowledge Distillation Beyond Model Compressioncited by 15
  19. 2018Rate Distortion For Model Compression: From Theory To Practicecited by 14
  20. 2019CAT: Compression-Aware Training for bandwidth reductioncited by 8
  21. 2019Learning Low-Rank Approximation for CNNscited by 7
  22. 2020Gaussian Approximation of Quantization Error for Estimation from Compressed Datacited by 3
  23. 2020How to Train your DNN: The Network Operator Editioncited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesiscited by 7