mlsanityData to 14 Oct 2026

Progressive Multi-Ancestor Bit-Depth Distillation

arXiv 2610.04100 ↗First posted 2 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Model compression strategies are widely employed to reduce memory footprint and network complexity, particularly for devices with constrained computational, memory, and energy resources. Prior works that rely on simultaneous conversion from floating-point high-precision (FP32) to integer low-precision (INT4) representations and distillation into smaller models suffer from unstable training and drastic degradation of prediction performance. To address these limitations, we propose a unified framework, known as Progressive Multi-Ancestor Bit-depth Distillation (PMABD), that progressively compresses the network while transferring knowledge through a growing pool of higher-precision ancestor teachers. PMABD generates a sequence of intermediate teachers that each learn from all higher-precision ancestors and jointly supervise the final target student. This multi-ancestor, multi-stage design stabilizes ultra-low-bit quantization by lowering quantization noise profiles across training and ensuring stable quantization. Experiments on CIFAR-10/100 with ResNet-20/32/18, and Tiny-ImageNet with MobileNetV2 show that PMABD outperforms state-of-the-art compression frameworks, results in 1.06% increase in performance of W2A2 (ResNet-18/CIFAR-100) student model. We show that a saturation-based stopping criterion contributes to improve the performance of our final student.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 30 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

26 of 30 references.

References · 26

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  2. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  5. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  6. 2019Relational Knowledge Distillationcited by 889
  7. 2019Learned Step Size Quantizationcited by 628
  8. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  9. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  10. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 101
  11. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  12. 2017Towards Effective Low-bitwidth Convolutional Neural Networkscited by 78
14 more
  1. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 72
  2. 2019QKD: Quantization-aware Knowledge Distillationcited by 44
  3. 2022Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networkscited by 23
  4. 2020Channel Distillation: Channel-Wise Attention for Knowledge Distillationcited by 21
  5. 2020Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networkscited by 21
  6. 2019Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activationscited by 18
  7. 2023Quantized Feature Distillation for Network Quantizationcited by 15
  8. 2023Poster: Self-Supervised Quantization-Aware Knowledge Distillationcited by 11
  9. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMscited by 8
  10. 2020Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networkscited by 7
  11. 2019Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networkscited by 5
  12. 2021InstantNet: Automated Generation and Deployment of Instantaneously Switchable-Precision Networkscited by 2
  13. 2025Data-Augmented Quantization-Aware Knowledge Distillationcited by 1
  14. 2025HPM-KD: Hierarchical Progressive Multi-Teacher Framework for Knowledge Distillation and Efficient Model Compressioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient training for compact compression models via sequential distillation
  2. 2026Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation
  3. 2024decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points
  4. 2025PQCAD-DM: Progressive Quantization and Calibration-Assisted Distillation for Extremely Efficient Diffusion Model
  5. 2026Sparse Knowledge Distillation: A Mathematical Framework for Probability-Domain Temperature Scaling and Multi-Stage Compression
  6. 2026Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression
  7. 2024Retraining-free Model Quantization via One-Shot Weight-Coupling Learning
  8. 2026Tree-VQ: Progressive Image Compression from Pretrained Vector Quantizers