mlsanityData to 14 Oct 2026

Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizer

arXiv 2509.20854 ↗First posted 25 Sep 2025cs.CVcited by 2

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Quantization-aware training (QAT) combined with knowledge distillation (KD) is a promising strategy for compressing Artificial Intelligence (AI) models for deployment on resource-constrained hardware. However, existing QAT-KD methods often struggle to balance task-specific (TS) and distillation losses due to heterogeneous gradient magnitudes, especially under low-bit quantization. We propose Game of Regularizer (GoR), a novel learnable regularization method that adaptively balances TS and KD objectives using only two trainable parameters for dynamic loss weighting. GoR reduces conflict between supervision signals, improves convergence, and boosts the performance of small quantized models (SQMs). Experiments on image classification, object detection (OD), and large language model (LLM) compression show that GoR consistently outperforms state-of-the-art QAT-KD methods. On low-power edge devices, it delivers faster inference while maintaining full-precision accuracy. We also introduce QAT-EKD-GoR, an ensemble distillation framework that uses multiple heterogeneous teacher models. Under optimal conditions, the proposed EKD-GoR can outperform full-precision models, providing a robust solution for real-world deployment.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 26 of its 36 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

26 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs19 Feb
  2. 2023Poster: Self-Supervised Quantization-Aware Knowledge Distillation22 Sep
  3. 2023Quantized Feature Distillation for Network Quantization20 Jul

Other references · 23

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  6. 2021YOLOX: Exceeding YOLO Series in 2021cited by 1,071
  7. 2020Designing Network Design Spacescited by 960
  8. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  9. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  10. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  11. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  12. 2020Channel-wise Knowledge Distillation for Dense Predictioncited by 163
11 more
  1. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 160
  2. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 119
  3. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 101
  4. 2017Towards Effective Low-bitwidth Convolutional Neural Networkscited by 78
  5. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 72
  6. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 40
  7. 2021Confidence-Aware Multi-Teacher Knowledge Distillationcited by 39
  8. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  9. 2022Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networkscited by 23
  10. 2025Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformerscited by 23
  11. 2020Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networkscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Gated Relational Alignment via Confidence-based Distillation for Efficient VLMscited by 6
  2. 2026Decoder-Free Distillation for Quantized Image Restorationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024In-Distribution Consistency Regularization Improves the Generalization of Quantization-Aware Training
  2. 2025Data-Augmented Quantization-Aware Knowledge Distillation
  3. 2024Self-Supervised Quantization-Aware Knowledge Distillation
  4. 2024QGen: On the Ability to Generalize in Quantization Aware Training
  5. 2026A Target-Centric Survey of Quantization-Aware Training
  6. 2025Regularization-based Framework for Quantization-, Fault- and Variability-Aware Training
  7. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery
  8. 2026What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Study