mlsanityData to 14 Oct 2026

Curriculum Temperature for Knowledge Distillation

arXiv 2211.16231 ↗First posted 29 Nov 2022cs.CVcited by 112

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Most existing distillation methods ignore the flexible role of the temperature in the loss function and fix it as a hyper-parameter that can be decided by an inefficient grid search. In general, the temperature controls the discrepancy between two distributions and can faithfully determine the difficulty level of the distillation task. Keeping a constant temperature, i.e., a fixed level of task difficulty, is usually sub-optimal for a growing student during its progressive learning stages. In this paper, we propose a simple curriculum-based technique, termed Curriculum Temperature for Knowledge Distillation (CTKD), which controls the task difficulty level during the student's learning career through a dynamic and learnable temperature. Specifically, following an easy-to-hard curriculum, we gradually increase the distillation loss w.r.t. the temperature, leading to increased distillation difficulty in an adversarial manner. As an easy-to-use plug-in technique, CTKD can be seamlessly integrated into existing knowledge distillation frameworks and brings general improvements at a negligible additional computation cost. Extensive experiments on CIFAR-100, ImageNet-2012, and MS-COCO demonstrate the effectiveness of our method. Our code is available at https://github.com/zhengli97/CTKD.

Key facts. Cited by 112 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 47 of its 60 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

47 of 60 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?29 Jun
  2. 2022Decoupled Knowledge Distillation16 Mar
  3. 2022Meta Knowledge Distillation16 Feb
  4. 2021Knowledge Distillation via Instance-level Sequence Learning21 Jun
  5. 2021Distilling Knowledge via Knowledge Review19 Apr

Other references · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  10. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  11. 2016Wide Residual Networkscited by 3,070
  12. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
30 more
  1. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  3. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  4. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  5. 2019Relational Knowledge Distillationcited by 889
  6. 2019Contrastive Representation Distillationcited by 748
  7. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
  8. 2019Similarity-Preserving Knowledge Distillationcited by 516
  9. 2020Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detectioncited by 409
  10. 2019Variational Information Distillation for Knowledge Transfercited by 324
  11. 2019Correlation Congruence for Knowledge Distillationcited by 285
  12. 2019Structured Knowledge Distillation for Dense Predictioncited by 279
  13. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  14. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 202
  15. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
  16. 2020Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classificationcited by 175
  17. 2019Online Knowledge Distillation with Diverse Peerscited by 128
  18. 2020Generalized Focal Loss V2: Learning Reliable Localization Quality Estimation for Dense Object Detectioncited by 104
  19. 2019Knowledge Distillation via Route Constrained Optimizationcited by 75
  20. 2020Curriculum DeepSDFcited by 62
  21. 2019Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narrativescited by 59
  22. 2021Online Knowledge Distillation for Efficient Pose Estimationcited by 55
  23. 2017Curriculum Dropoutcited by 50
  24. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
  25. 2018Learning to Teach with Dynamic Loss Functionscited by 40
  26. 2021Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitationcited by 40
  27. 2019Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and Morecited by 29
  28. 2019Curriculum-based transfer learning for an effective end-to-end spoken language understanding and domain portabilitycited by 29
  29. 2020Online Knowledge Distillation via Multi-branch Diversity Enhancementcited by 21
  30. 2020An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 14; 2024: 38; 2025: 36; 2026: 23.

112 papers.

Sort:

By area:

  1. 2024Logit Standardization in Knowledge Distillationcited by 102
  2. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  3. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  4. 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 55
  5. 2023CrossKD: Cross-Head Knowledge Distillation for Object Detectioncited by 52
  6. 2024Knowledge Distillation Based on Transformed Teacher Matchingcited by 37
  7. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 29
  8. 2024Text2LiDAR: Text-guided LiDAR Point Cloud Generation via Equirectangular Transformercited by 27
  9. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  10. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 22
  11. 2023Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?cited by 21
  12. 2024Advancing Textual Prompt Learning with Anchored Attributescited by 20
38 more
  1. 2024Cascade Prompt Learning for Vision-Language Model Adaptationcited by 19
  2. 2024Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillationcited by 19
  3. 2023SeaFormer++: Squeeze-enhanced Axial Transformer for Mobile Visual Recognitioncited by 18
  4. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  5. 2026Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoningcited by 17
  6. 2024A Review of Pseudo-Labeling for Computer Visioncited by 16
  7. 2026Adaptive Teacher Exposure for Self-Distillation in LLM Reasoningcited by 14
  8. 2023Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillationcited by 13
  9. 2025LLM-Oriented Token-Adaptive Knowledge Distillationcited by 13
  10. 2023Curricular Object Manipulation in LiDAR-based Object Detectioncited by 12
  11. 2024AMD: Automatic Multi-step Distillation of Large-scale Vision Modelscited by 12
  12. 2023GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Rankingcited by 11
  13. 2024To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DROcited by 11
  14. 2025Rapid Salient Object Detection with Difference Convolutional Neural Networkscited by 10
  15. 2023SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Datacited by 9
  16. 2024Knowledge Distillation with Refined Logitscited by 9
  17. 2025Decoupled Global-Local Alignment for Improving Compositional Understandingcited by 9
  18. 2023The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformerscited by 8
  19. 2023EA-KD: Entropy-based Adaptive Knowledge Distillationcited by 8
  20. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 7
  21. 2024Dynamic Temperature Knowledge Distillationcited by 6
  22. 2024DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learningcited by 6
  23. 2025VRM: Knowledge Distillation via Virtual Relation Matchingcited by 6
  24. 2024Improve Knowledge Distillation via Label Revision and Data Selectioncited by 5
  25. 2025Perspective-Aware Teaching: Adapting Knowledge for Heterogeneous Distillationcited by 5
  26. 2025Test-Time Distillation for Continual Model Adaptationcited by 5
  27. 2025DGS-Net: Distillation-Guided Gradient Surgery for CLIP Fine-Tuning in AI-Generated Image Detectioncited by 5
  28. 2025Sample-level Adaptive Knowledge Distillation for Action Recognitioncited by 4
  29. 2025DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainercited by 4
  30. 2025Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Frameworkcited by 4
  31. 2025Competitive Distillation: A Simple Learning Strategy for Improving Visual Classificationcited by 4
  32. 2025Local Dense Logit Relations for Enhanced Knowledge Distillationcited by 4
  33. 2025Cross-Architecture Distillation Made Simple with Redundancy Suppressioncited by 4
  34. 2025NAIPv2: Debiased Pairwise Learning for Efficient Paper Quality Estimationcited by 4
  35. 2023Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approachcited by 3
  36. 2024Attention-guided Feature Distillation for Semantic Segmentationcited by 3
  37. 2024CNN2GNN: How to Bridge CNN with GNNcited by 3
  38. 2024AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weightingcited by 3

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again
  2. 2023Student-friendly Knowledge Distillation
  3. 2023A Survey on Recent Teacher-student Learning Studies
  4. 2025Efficient Knowledge Distillation via Curriculum Extraction
  5. 2023Can a student Large Language Model perform as well as it's teacher?
  6. 2025UNDO: Understanding Distillation as Optimization
  7. 2026Distillation Traps and Guards: A Calibration Knob for LLM Distillability
  8. 2022Parameter-Efficient and Student-Friendly Knowledge Distillation