mlsanityData to 14 Oct 2026

Close Shortcut Wins Long: Seeking Diverse and Stable Generators for Data-Free Knowledge Distillation

arXiv 2608.22003 ↗First posted 22 Aug 2026cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Data-Free Knowledge Distillation (DFKD) preserves privacy by transferring knowledge without real data access. However, existing generator-based DFKD methods suffer from over-reliance on teacher preferences and pattern collapse, exhibiting "generative shortcut learning" in the frequency domain: dependent on specific frequency components and frequency positions, resulting in inconsistent synthetic image quality and class diversity. In this paper, we propose a CSWL framework aimed at introducing insights from the frequency domain perspective to improve generator diversity and training stability to Close the phenomenon of Shortcut learning to Win in the Longer term. To address the issue of generative shortcut learning, we introduce frequency-domain augmentation at the feature level, encouraging the generator to attend to the full frequency spectrum and thereby suppress shortcut learning behavior. To tackle training instability, we propose a Cross-Stage Frequency Reconstruction (CSFR) auxiliary task, which implicitly constructs an Exponential Moving Average (EMA) mechanism to promote long-term optimization and stability. Extensive experiments, including downstream tasks and various image recognition datasets at multiple resolutions, validate the effectiveness of CSWL in improving both diversity and stability from the frequency view.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 47 of its 60 references that are in the map. First posted on arXiv 22 Aug 2026.

Builds on

47 of 60 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning16 Sep
  2. 2023FreeKD: Knowledge Distillation via Semantic Frequency Prompt20 Nov
  3. 2023Customizing Synthetic Data for Data-Free Student Learning10 Jul
  4. 2023Class Attention Transfer Based Knowledge Distillation25 Apr
  5. 2023Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation28 Feb
  6. 2022Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation21 Sep
  7. 2021Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data11 Aug
  8. 2021Contrastive Model Inversion for Data-Free Knowledge Distillation18 May

Other references · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2016Wide Residual Networkscited by 3,070
  6. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  8. 2017Mixed Precision Trainingcited by 1,156
  9. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  10. 2020Dataset Condensation with Gradient Matchingcited by 516
  11. 2013Do Deep Nets Really Need to be Deep?cited by 474
  12. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
27 more
  1. 2021Dataset Condensation with Distribution Matchingcited by 365
  2. 2021Spectral Temporal Graph Neural Network for Multivariate Time-series Forecastingcited by 299
  3. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 291
  4. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  5. 2018Dataset Distillationcited by 230
  6. 2016Unrolled Generative Adversarial Networkscited by 208
  7. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  8. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  9. 2019Zero-Shot Knowledge Distillation in Deep Networkscited by 121
  10. 2019Data-Free Adversarial Distillationcited by 90
  11. 2020Data-Free Network Quantization With Adversarial Knowledge Distillationcited by 80
  12. 2018The Unusual Effectiveness of Averaging in GAN Trainingcited by 73
  13. 2021Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domaincited by 71
  14. 2022Delving into Effective Gradient Matching for Dataset Condensationcited by 40
  15. 2023X3KD: Knowledge Distillation Across Modalities, Tasks and Stages for Multi-Camera 3D Object Detectioncited by 28
  16. 2023What do neural networks learn in image classification? A frequency shortcut perspectivecited by 28
  17. 2023Adaptive Data-Free Quantizationcited by 26
  18. 2023NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillationcited by 22
  19. 2023Hard Sample Matters a Lot in Zero-Shot Quantizationcited by 15
  20. 2022FAKD: Feature Augmented Knowledge Distillation for Semantic Segmentationcited by 14
  21. 2021Exploring The Effect of High-frequency Components in GANs Trainingcited by 10
  22. 2020Robustness and Diversity Seeking Data-Free Knowledge Distillationcited by 9
  23. 2026HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory Systemcited by 8
  24. 2026Dual-Anchoring: Addressing State Drift in Vision-Language Navigationcited by 7
  25. 2022A Challenging Benchmark of Anime Style Recognitioncited by 6
  26. 2025TouchFormer: A Robust Transformer-based Framework for Multimodal Material Perceptioncited by 4
  27. 2025ClearGCD: Mitigating Shortcut Learning For Robust Generalized Category Discoverycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation
  2. 2021Conditional Generative Data-free Knowledge Distillation
  3. 2026Diverse Image Priors for Black-box Data-free Knowledge Distillation
  4. 2025CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillation
  5. 2024Small Scale Data-Free Knowledge Distillation
  6. 2022Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay
  7. 2025PRISM: Precision-Recall Informed Data-Free Knowledge Distillation via Generative Diffusion
  8. 2024Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architectures