mlsanityData to 14 Oct 2026

MEC-Quant: Maximum Entropy Coding for Extremely Low Bit Quantization-Aware Training

arXiv 2509.15514 ↗First posted 19 Sep 2025cs.CVcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Quantization-Aware Training (QAT) has driven much attention to produce efficient neural networks. Current QAT still obtains inferior performances compared with the Full Precision (FP) counterpart. In this work, we argue that quantization inevitably introduce biases into the learned representation, especially under the extremely low-bit setting. To cope with this issue, we propose Maximum Entropy Coding Quantization (MEC-Quant), a more principled objective that explicitly optimizes on the structure of the representation, so that the learned representation is less biased and thus generalizes better to unseen in-distribution samples. To make the objective end-to-end trainable, we propose to leverage the minimal coding length in lossy data coding as a computationally tractable surrogate for the entropy, and further derive a scalable reformulation of the objective based on Mixture Of Experts (MOE) that not only allows fast computation but also handles the long-tailed distribution for weights or activation values. Extensive experiments on various tasks on computer vision tasks prove its superiority. With MEC-Qaunt, the limit of QAT is pushed to the x-bit activation for the first time and the accuracy of MEC-Quant is comparable to or even surpass the FP counterpart. Without bells and whistles, MEC-Qaunt establishes a new state of the art for QAT.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 28 of its 33 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

28 of 33 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  8. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  9. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  11. 2019Learned Step Size Quantizationcited by 628
  12. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
16 more
  1. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  2. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  3. 2018Model compression via distillation and quantizationcited by 289
  4. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  5. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  6. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  7. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 108
  8. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  9. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 79
  10. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 45
  11. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 40
  12. 2022Self-Supervised Learning via Maximum Entropy Codingcited by 33
  13. 2023Q-DETR: An Efficient Low-Bit Quantized Detection Transformercited by 24
  14. 2018Quantization Mimic: Towards Very Tiny CNN for Object Detectioncited by 20
  15. 2023Resilient Binary Neural Networkcited by 13
  16. 2022Synergistic Self-supervised and Quantization Learningcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing Learned Image Compression on Scalar and Entropy-Constraint Quantization
  2. 2025Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding
  3. 2023Optimized learned entropy coding parameters for practical neural-based image and video compression
  4. 2025Exploiting Latent Properties to Optimize Neural Codecs
  5. 2022Bitwidth-Adaptive Quantization-Aware Neural Network Training: A Meta-Learning Approach
  6. 2026Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression
  7. 2021Soft then Hard: Rethinking the Quantization in Neural Image Compression
  8. 2024Efficient Neural Compression with Inference-time Decoding