mlsanityData to 7 Oct 2026

Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks

arXiv 2410.02116 ↗First posted 3 Oct 2024cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Dataset distillation (DD) generates small synthetic datasets that can efficiently train deep networks with a limited amount of memory and compute. Despite the success of DD methods for supervised learning, DD for self-supervised pre-training of deep models has remained unaddressed. Pre-training on unlabeled data is crucial for efficiently generalizing to downstream tasks with limited labeled data. In this work, we propose the first effective DD method for SSL pre-training. First, we show, theoretically and empirically, that naive application of supervised DD methods to SSL fails, due to the high variance of the SSL gradient. Then, we address this issue by relying on insights from knowledge distillation (KD) literature. Specifically, we train a small student model to match the representations of a larger teacher model trained with SSL. Then, we generate a small synthetic dataset by matching the training trajectories of the student models. As the KD objective has considerably lower variance than SSL, our approach can generate synthetic datasets that can successfully pre-train high-quality encoders. Through extensive experiments, we show that our distilled sets lead to up to 13% higher accuracy than prior work, on a variety of downstream tasks, in the presence of limited labeled data. Code at https://github.com/BigML-CS-UCLA/MKDT.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 40 of its 48 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

40 of 48 references.

References · 40

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  5. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  6. 2019Relational Knowledge Distillationcited by 873
  7. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  8. 2020Contrastive Learning with Hard Negative Samplescited by 619
  9. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  10. 2020Dataset Condensation with Gradient Matchingcited by 510
  11. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  12. 2021Dataset Condensation with Distribution Matchingcited by 362
28 more
  1. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  2. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  3. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  4. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  5. 2022Knowledge Distillation from A Stronger Teachercited by 204
  6. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 187
  7. 2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 180
  8. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  9. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  10. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  11. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 136
  12. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  13. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 119
  14. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  15. 2022Dataset Condensation with Contrastive Signalscited by 105
  16. 2023DataDAM: Efficient Dataset Distillation with Attention Matchingcited by 83
  17. 2020Variance-Reduced Methods for Machine Learningcited by 79
  18. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  19. 2023Simple and Fast Group Robustness by Automatic Feature Reweightingcited by 68
  20. 2020CompRess: Self-Supervised Learning by Compressing Representationscited by 61
  21. 2017DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfercited by 55
  22. 2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
  23. 2019Learning Metrics from Teachers: Compact Networks for Image Embeddingcited by 38
  24. 2023Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppressioncited by 30
  25. 2023Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Leastcited by 22
  26. 2024Data-Efficient Contrastive Language-Image Pretraining: Prioritizing Data Quality over Quantitycited by 19
  27. 2023Data Distillation Can Be Like Vodka: Distilling More Times For Better Qualitycited by 17
  28. 2023Guarding Barlow Twins Against Overfitting with Mixed Samplescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutionscited by 24
  2. 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluationscited by 7
  3. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alonecited by 2
  4. 2025EEG-DLite: Dataset Distillation for Efficient Large EEG Model Trainingcited by 0
  5. 2026Omnimodal Dataset Distillation via High-order Proxy Alignmentcited by 0
  6. 2026Text-attributed Graph Condensation via Text Selection and Attribute Matchingcited by 0