mlsanityData to 14 Oct 2026

IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves

arXiv 2608.24759 ↗First posted 25 Aug 2026cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Multi-teacher distillation has emerged as a way to combine complementary teacher models into a single student model that exhibits the strengths of all its teachers. The student is trained to mimic the output of the teachers on a set of images, typically the union of the individual teacher's training sets, assuming this data is available. In this paper, we question that assumption and explore alternative options. We first study how far one can go when distilling from teachers fed with different types of noise. Then, we show that information contained in the teachers can be leveraged to tailor the noise for multi-teacher distillation: we propose a method that, thanks to decorrelation losses at both patch and image levels, generates teacher-specific, improved samples optimized for data-free distillation. Experiments show that our most effective samples, IDeaL, lead to strong students that successfully capture complementary information from the teachers, yielding surprisingly competitive results that substantially narrow the gap with students distilled from real images. Moreover, given a limited budget of 1K images for distillation, students distilled using our IDeaL samples match or surpass the performance of those distilled using a 1K-image subset of ImageNet.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 61 of its 77 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

61 of 77 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Cross-Architecture Distillation Made Simple with Redundancy Suppression29 Jul
  2. 2024What Makes a Good Dataset for Knowledge Distillation?19 Nov
  3. 2024UNIC: Universal Classification Models via Multi-teacher Distillation9 Aug
  4. 2024UDON: Universal Dynamic Online distillatioN for generic image representations12 Jun
  5. 2023Data-Free Knowledge Distillation Using Adversarially Perturbed OpenGL Shader Images20 Oct
  6. 2023Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation28 Feb
  7. 2022Exploring Target Representations for Masked Autoencoders8 Sep
  8. 2021Contrastive Model Inversion for Data-Free Knowledge Distillation18 May

Other references · 53

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2013Describing Textures in the Wildcited by 2,125
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
38 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  2. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  4. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  5. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  6. 2019Contrastive Representation Distillationcited by 748
  7. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  8. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  9. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  10. 2022DeiT III: Revenge of the ViTcited by 419
  11. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
  12. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  13. 2019Data-Free Learning of Student Networkscited by 204
  14. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 172
  15. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  16. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  17. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  18. 2019Zero-Shot Knowledge Distillation in Deep Networkscited by 121
  19. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 108
  20. 2021Pre-training without Natural Imagescited by 100
  21. 2021Learning to See by Looking at Noisecited by 92
  22. 2019Data-Free Adversarial Distillationcited by 90
  23. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 89
  24. 2021Diversifying Sample Generation for Accurate Data-Free Quantizationcited by 63
  25. 2022Patch Similarity Aware Data-Free Quantization for Vision Transformerscited by 56
  26. 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
  27. 2020Concept Generalization in Visual Representation Learningcited by 35
  28. 2020Data-Free Knowledge Amalgamation via Group-Stack Dual-GANcited by 32
  29. 2021Can Vision Transformers Learn without Natural Images?cited by 32
  30. 2025DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teacherscited by 32
  31. 2023Visual Atoms: Pre-training Vision Transformers with Sinusoidal Wavescited by 20
  32. 2023Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Modelcited by 19
  33. 2024CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTscited by 17
  34. 2024Scaling Backwards: Minimal Synthetic Pre-training?cited by 14
  35. 2023Pre-training Vision Transformers with Very Limited Synthesized Imagescited by 12
  36. 2023A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptationcited by 11
  37. 2024MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similaritycited by 9
  38. 2023Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learnerscited by 6

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Level Feature Distillation of Joint Teachers Trained on Distinct Image Datasets
  2. 2021Teacher's pet: understanding and mitigating biases in distillation
  3. 2024Multi-student Diffusion Distillation for Better One-step Generators
  4. 2022Teaching What You Should Teach: A Data-Based Distillation Method
  5. 2021Fixing the Teacher-Student Knowledge Discrepancy in Distillation
  6. 2025Distillation Scaling Laws
  7. 2023On student-teacher deviations in distillation: does it pay to disobey?
  8. 2024PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation