mlsanityData to 14 Oct 2026

StableKD: Breaking Inter-block Optimization Entanglement for Stable Knowledge Distillation

arXiv 2312.13223 ↗First posted 20 Dec 2023cs.CVcited by 0

On the map: Knowledge distillation → Graph neural network distillation

Abstract (arXiv)

Knowledge distillation (KD) has been recognized as an effective tool to compress and accelerate models. However, current KD approaches generally suffer from an accuracy drop and/or an excruciatingly long distillation process. In this paper, we tackle the issue by first providing a new insight into a phenomenon that we call the Inter-Block Optimization Entanglement (IBOE), which makes the conventional end-to-end KD approaches unstable with noisy gradients. We then propose StableKD, a novel KD framework that breaks the IBOE and achieves more stable optimization. StableKD distinguishes itself through two operations: Decomposition and Recomposition, where the former divides a pair of teacher and student networks into several blocks for separate distillation, and the latter progressively merges them back, evolving towards end-to-end distillation. We conduct extensive experiments on CIFAR100, Imagewoof, and ImageNet datasets with various teacher-student pairs. Compared to other KD approaches, our simple yet effective StableKD greatly boosts the model accuracy by 1% ~ 18%, speeds up the convergence up to 10 times, and outperforms them with only 40% of the training data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation. Builds on 38 of its 48 references that are in the map. First posted on arXiv 20 Dec 2023.

Builds on

38 of 48 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2016Wide Residual Networkscited by 3,070
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,653
26 more
  1. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  2. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  3. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  4. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  5. 2018Born Again Neural Networkscited by 546
  6. 2022Decoupled Knowledge Distillationcited by 388
  7. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  8. 2021Distilling Knowledge via Knowledge Reviewcited by 288
  9. 2019Correlation Congruence for Knowledge Distillationcited by 285
  10. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  11. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259
  12. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  13. 2017Sobolev Training for Neural Networkscited by 143
  14. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  15. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 118
  16. 2017Failures of Gradient-Based Deep Learningcited by 80
  17. 2021Fair Feature Distillation for Visual Recognitioncited by 59
  18. 2021Online Knowledge Distillation for Efficient Pose Estimationcited by 55
  19. 2021A Fast Knowledge Distillation Framework for Visual Recognitioncited by 50
  20. 2018Amalgamating Knowledge towards Comprehensive Classificationcited by 38
  21. 2020Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Modelcited by 28
  22. 2019Few Shot Network Compression via Cross Distillationcited by 26
  23. 2020Progressive Network Grafting for Few-Shot Knowledge Distillationcited by 25
  24. 2020Differentiable Feature Aggregation Search for Knowledge Distillationcited by 23
  25. 2022Black-box Few-shot Knowledge Distillationcited by 14
  26. 2020Data-Efficient Ranking Distillation for Image Retrievalcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Knowledge Distillation for Variational Quantum Convolutional Neural Networks on Heterogeneous Data
  2. 2023Bridging Classical and Quantum Machine Learning: Knowledge Transfer From Classical to Quantum Neural Networks Using Knowledge Distillation
  3. 2025KARIPAP: Quantum-Inspired Tensor Network Compression of Large Language Models Using Infinite Projected Entangled Pair States and Tensor Renormalization Group
  4. 2025DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer
  5. 2025Do GNN-based QEC Decoders Require Classical Knowledge? Evaluating the Efficacy of Knowledge Distillation from MWPM
  6. 2026The Distributional View of Knowledge Distillation
  7. 2024Enhancing Logits Distillation with Plug&Play Kendall's τ Ranking Loss
  8. 2024Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models