mlsanityData to 13 Oct 2026

Dataset Distillation

arXiv 1811.10959 ↗First posted 27 Nov 2018cs.LG stat.MLcited by 226

Not on the map. Why?

Abstract (arXiv)

Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few gradient descent steps, given a fixed network initialization. We evaluate our method in various initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods.

Key facts. Cited by 226 papers in the map (data through 13 Oct 2026). First posted on arXiv 27 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 17; 2022: 40; 2023: 41; 2024: 54; 2025: 38; 2026: 36.

226 papers.

Sort:

By area:

  1. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 730
  2. 2022Dataset Distillation by Matching Training Trajectoriescited by 438
  3. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  4. 2021Dataset Condensation with Distribution Matchingcited by 363
  5. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  6. 2021Efficient and Modular Implicit Differentiationcited by 201
  7. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  8. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 147
  9. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  10. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 137
  11. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  12. 2023Generalizing Dataset Distillation via Deep Generative Priorcited by 125
38 more
  1. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  2. 2023Improved Distribution Matching for Dataset Condensationcited by 124
  3. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  4. 2022Dataset Condensation with Contrastive Signalscited by 105
  5. 2022DC-BENCH: Dataset Condensation Benchmarkcited by 67
  6. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  7. 2023CaT: Balanced Continual Graph Learning with Graph Condensationcited by 52
  8. 2021Soft-Label Anonymous Gastric X-ray Image Distillationcited by 51
  9. 2021A Fast Knowledge Distillation Framework for Visual Recognitioncited by 50
  10. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  11. 2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
  12. 2021Federated Learning Meets Natural Language Processing: A Surveycited by 43
  13. 2022Delving into Effective Gradient Matching for Dataset Condensationcited by 40
  14. 2024Elucidating the Design Space of Dataset Condensationcited by 40
  15. 2023An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learningcited by 39
  16. 2021HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networkscited by 38
  17. 2022Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharingcited by 35
  18. 2022Private Set Generation with Discriminative Informationcited by 34
  19. 2022Structure-aware Protein Self-supervised Learningcited by 33
  20. 2024An Aggregation-Free Federated Learning for Tackling Data Heterogeneitycited by 33
  21. 2023Loss-Curvature Matching for Dataset Selection and Condensationcited by 32
  22. 2023A Survey on Dataset Distillation: Approaches, Applications and Future Directionscited by 32
  23. 2022Sample Condensation in Online Continual Learningcited by 31
  24. 2023Graph Condensation for Inductive Node Representation Learningcited by 31
  25. 2023Dataset Distillation via the Wasserstein Metriccited by 31
  26. 2024A Label is Worth a Thousand Images in Dataset Distillationcited by 31
  27. 2022FedSynth: Gradient Compression via Synthetic Data in Federated Learningcited by 30
  28. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  29. 2021Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Errorcited by 28
  30. 2025A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advancescited by 28
  31. 2022Dataset Distillation Using Parameter Pruningcited by 27
  32. 2022Gradient-based Bi-level Optimization for Deep Learning: A Surveycited by 26
  33. 2024Dataset Distillation by Automatic Training Trajectoriescited by 26
  34. 2022Dataset Distillation for Medical Dataset Sharingcited by 25
  35. 2025The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutionscited by 25
  36. 2023Backdoor Attacks Against Dataset Distillationcited by 21
  37. 2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiencycited by 21
  38. 2021Condensed Composite Memory Continual Learningcited by 20

50 most cited of 226 All 226 (JSON)