mlsanityData to 14 Oct 2026

Decoupled Kullback-Leibler Divergence Loss

arXiv 2305.13948 ↗First posted 23 May 2023cs.CV cs.LGcited by 58

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss and mathematically prove that it is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss that consists of 1) a weighted Mean Square Error (wMSE) loss and 2) a Cross-Entropy loss incorporating soft labels. Thanks to the decomposed formulation of DKL loss, we have identified two areas for improvement. Firstly, we address the limitation of KL/DKL in scenarios like knowledge distillation by breaking its asymmetric optimization property. This modification ensures that the wMSE component is always effective during training, providing extra constructive cues. Secondly, we introduce class-wise global information into KL/DKL to mitigate bias from individual samples. With these two enhancements, we derive the Improved Kullback-Leibler (IKL) Divergence loss and evaluate its effectiveness by conducting experiments on CIFAR-10/100 and ImageNet datasets, focusing on adversarial training, and knowledge distillation tasks. The proposed approach achieves new state-of-the-art adversarial robustness on the public leaderboard -- RobustBench and competitive performance on knowledge distillation, demonstrating the substantial practical merits. Our code is available at https://github.com/jiequancui/DKL.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 63 of its 79 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

63 of 79 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Knowledge Distillation from A Stronger Teacher21 May
  2. 2022Decoupled Knowledge Distillation16 Mar
  3. 2021Distilling Knowledge via Knowledge Review19 Apr

Other references · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2017Mask R-CNNcited by 7,277
  12. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
38 more
  1. 2013Intriguing properties of neural networkscited by 5,155
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  3. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  4. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  8. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  10. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  11. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  12. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  13. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  14. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  15. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  16. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  17. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  18. 2019Relational Knowledge Distillationcited by 889
  19. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
  20. 2017Focal Loss for Dense Object Detectioncited by 752
  21. 2019Contrastive Representation Distillationcited by 748
  22. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  23. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  24. 2017Deep Mutual Learningcited by 628
  25. 2020Long-tail learning via logit adjustmentcited by 618
  26. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  27. 2018Born Again Neural Networkscited by 546
  28. 2020Overfitting in adversarially robust deep learningcited by 506
  29. 2019On the Efficacy of Knowledge Distillationcited by 381
  30. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  31. 2019Correlation Congruence for Knowledge Distillationcited by 285
  32. 2020Equalization Loss for Long-Tailed Object Recognitioncited by 285
  33. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  34. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  35. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  36. 2021Improving Robustness using Generated Datacited by 234
  37. 2021Parametric Contrastive Learningcited by 217
  38. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 208

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 22; 2025: 18; 2026: 12.

58 papers.

Sort:

By area:

  1. 2023Boosting Few-shot 3D Point Cloud Segmentation via Query-Guided Enhancementcited by 19
  2. 2024Rethinking Centered Kernel Alignment in Knowledge Distillationcited by 17
  3. 2024MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifierscited by 16
  4. 2024Adversarial Training on Purification (AToP): Advancing Both Robustness and Generalizationcited by 15
  5. 2025AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defensescited by 15
  6. 2024Ensemble everything everywhere: Multi-scale aggregation for adversarial robustnesscited by 12
  7. 2024DAT: Improving Adversarial Robustness via Generative Amplitude Mix-up in Frequency Domaincited by 12
  8. 2024Classes Are Not Equal: An Empirical Study on Image Recognition Fairnesscited by 10
  9. 2024Detecting Brittle Decisions for Free: Leveraging Margin Consistency in Deep Robust Classifierscited by 10
  10. 2024Shedding More Light on Robust Classifiers under the lens of Energy-based Modelscited by 9
  11. 2024Adversarial Guided Diffusion Models for Adversarial Purificationcited by 8
  12. 2023SWAP: Exploiting Second-Ranked Logits for Adversarial Attacks on Time Seriescited by 6
38 more
  1. 2023Assessing Robustness via Score-Based Adversarial Image Generationcited by 6
  2. 2023Indirect Gradient Matching for Adversarial Robust Distillationcited by 6
  3. 2025Generalized Kullback-Leibler Divergence Losscited by 6
  4. 2026Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillationcited by 5
  5. 2023OODRobustBench: a Benchmark and Large-Scale Analysis of Adversarial Robustness under Distribution Shiftcited by 4
  6. 2024Sparse-PGD: A Unified Framework for Sparse Adversarial Perturbations Generationcited by 4
  7. 2025Implicit Inversion turns CLIP into a Decodercited by 4
  8. 2024A Random Ensemble of Encrypted Vision Transformers for Adversarially Robust Defensecited by 3
  9. 2024Data-Driven Lipschitz Continuity: A Cost-Effective Approach to Improve Adversarial Robustnesscited by 3
  10. 2024Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attackscited by 3
  11. 2024On the Robustness of Adversarial Training Against Uncertainty Attackscited by 2
  12. 2025Understanding Adversarial Training with Energy-based Modelscited by 2
  13. 2025DASH: A Meta-Attack Framework for Synthesizing Effective and Stealthy Adversarial Examplescited by 2
  14. 2025Robustness Feature Adapter for Efficient Adversarial Trainingcited by 2
  15. 2023A Comprehensive Assessment Benchmark for Rigorously Evaluating Deep Learning Image Classifierscited by 1
  16. 2024ZeroPur: Succinct Training-Free Adversarial Purificationcited by 1
  17. 2024MALT Powers Up Adversarial Attackscited by 1
  18. 2024Optimizing Vision Transformers with Data-Free Knowledge Transfercited by 1
  19. 2024Test-time Adversarial Defense with Opposite Adversarial Path and High Attack Time Costcited by 1
  20. 2025Wasserstein distributional adversarial training for deep neural networkscited by 1
  21. 2025Deep Positive-Negative Prototypes for Adversarially Robust Discriminative Prototypical Learningcited by 1
  22. 2025Generative Distribution Distillationcited by 1
  23. 2025Rejuvenating Cross-Entropy Loss in Knowledge Distillation for Recommender Systemscited by 1
  24. 2025Calibrating Uncertainty for Zero-Shot Adversarial CLIPcited by 1
  25. 2024On adversarial training and the 1 Nearest Neighbor classifiercited by 0
  26. 2024Enhancing Output Diversity Improves Conjugate Gradient-based Adversarial Attackscited by 0
  27. 2024Environment Maps Editing using Inverse Rendering and Adversarial Implicit Functionscited by 0
  28. 2024Never Reset Again: A Mathematical Framework for Continual Inference in Recurrent Neural Networkscited by 0
  29. 2025Diffusion or Non-Diffusion Adversarial Defenses: Rethinking the Relation between Classifier and Adversarial Purifiercited by 0
  30. 2025Model-Free Adversarial Purification via Coarse-To-Fine Tensor Network Representationcited by 0
  31. 2025Fast Adversarial Training against Sparse Attacks Requires Loss Smoothingcited by 0
  32. 2025Feature Statistics with Uncertainty Help Adversarial Robustnesscited by 0
  33. 2025RegMix: Adversarial Mutual and Generalization Regularization for Enhancing DNN Robustnesscited by 0
  34. 2025Tight Robustness Certificates and Wasserstein Distributional Attacks for Deep Neural Networkscited by 0
  35. 2025Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillationcited by 0
  36. 2026Integrating Knowledge Distillation Methods: A Sequential Multi-Stage Frameworkcited by 0
  37. 2026Expanding the Role of Diffusion Models for Robust Classifier Trainingcited by 0
  38. 2026Why the Maximum Second Derivative of Activations Matters for Adversarial Robustnesscited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
  2. 2024Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matching
  3. 2025Balance Divergence for Knowledge Distillation
  4. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
  5. 2022Blind Knowledge Distillation for Robust Image Classification
  6. 2024Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation
  7. 2025Better Estimation of the Kullback--Leibler Divergence Between Language Models
  8. 2026Optimal Stability of KL Divergence under Gaussian Perturbations