mlsanityData to 7 Oct 2026

Data Distillation: A Survey

arXiv 2301.04272 ↗First posted 11 Jan 2023cs.LG cs.CV cs.IRcited by 86

Not on the map. Why?

Abstract (arXiv)

The popularity of deep learning has led to the curation of a vast number of massive and multifarious datasets. Despite having close-to-human performance on individual tasks, training parameter-hungry models on large datasets poses multi-faceted problems such as (a) high model-training time; (b) slow research iteration; and (c) poor eco-sustainability. As an alternative, data distillation approaches aim to synthesize terse data summaries, which can serve as effective drop-in replacements of the original dataset for scenarios like model training, inference, architecture search, etc. In this survey, we present a formal framework for data distillation, along with providing a detailed taxonomy of existing approaches. Additionally, we cover data distillation approaches for different data modalities, namely images, graphs, and user-item interactions (recommender systems), while also identifying current challenges and future research directions.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Builds on 106 of its 131 references that are in the map. First posted on arXiv 11 Jan 2023.

Builds on

106 of 131 references.

References · 106

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  10. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
38 more
  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  8. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2018Self-Attentive Sequential Recommendationcited by 1,169
  11. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  12. 2019Graph Neural Networks for Social Recommendationcited by 711
  13. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  14. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  15. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  16. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  17. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  18. 2020Dataset Condensation with Gradient Matchingcited by 510
  19. 2020Graph Neural Networks in Recommender Systems: A Surveycited by 493
  20. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  21. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  22. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 403
  23. 2016Accelerating the Super-Resolution Convolutional Neural Networkcited by 393
  24. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  25. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 377
  26. 2021Dataset Condensation with Distribution Matchingcited by 362
  27. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  28. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  29. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  30. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 326
  31. 2018Session-based Recommendation with Graph Neural Networkscited by 312
  32. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  33. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  34. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
  35. 2016Recommendations as Treatments: Debiasing Learning and Evaluationcited by 246
  36. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  37. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  38. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 34; 2025: 21; 2026: 10.

86 papers.

Sort:

By area:

  1. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 136
  2. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  3. 2024How to Train Data-Efficient LLMscited by 91
  4. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 76
  5. 2023Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Datacited by 71
  6. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  7. 2024A Comprehensive Survey on Graph Reduction: Sparsification, Coarsening, and Condensationcited by 57
  8. 2023Sequential Subset Matching for Dataset Distillationcited by 40
  9. 2024Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustmentcited by 32
  10. 2024A Label is Worth a Thousand Images in Dataset Distillationcited by 31
  11. 2023Frequency Domain-based Dataset Distillationcited by 28
  12. 2024CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Modelscited by 26
38 more
  1. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 26
  2. 2024Graph Condensation: A Surveycited by 23
  3. 2023Vision-Language Dataset Distillationcited by 18
  4. 2024Importance-Aware Adaptive Dataset Distillationcited by 18
  5. 2024Towards Adversarially Robust Dataset Distillation by Curvature Regularizationcited by 17
  6. 2025Dataset Distillation via Vision-Language Category Prototypecited by 17
  7. 2023Towards Trustworthy Dataset Distillationcited by 14
  8. 2024Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillationcited by 14
  9. 2024Self-supervised Dataset Distillation: A Good Compression Is All You Needcited by 13
  10. 2024Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensatorcited by 13
  11. 2024Generative Dataset Distillation Based on Diffusion Modelcited by 13
  12. 2025TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendationcited by 13
  13. 2025Information-Guided Diffusion Sampling for Dataset Distillationcited by 10
  14. 2024Distilling Long-tailed Datasetscited by 9
  15. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  16. 2023Dataset Distillation in Latent Spacecited by 7
  17. 2024Improve Cross-Architecture Generalization on Dataset Distillationcited by 7
  18. 2024Distributional Dataset Distillation with Subtask Decompositioncited by 7
  19. 2024GC4NC: A Benchmark Framework for Graph Condensation on Node Classification with New Insightscited by 7
  20. 2023Federated Learning on Virtual Heterogeneous Data with Local-global Distillationcited by 6
  21. 2023DREAM+: Efficient Dataset Distillation by Bidirectional Representative Matchingcited by 6
  22. 2023Data Optimization in Deep Learning: A Surveycited by 6
  23. 2024Dataset Condensation with Latent Quantile Matchingcited by 6
  24. 2023Multi-Source Domain Adaptation meets Dataset Distillation through Dataset Dictionary Learningcited by 5
  25. 2023Embarassingly Simple Dataset Distillationcited by 5
  26. 2024Progressive trajectory matching for medical dataset distillationcited by 5
  27. 2024Behaviour Distillationcited by 5
  28. 2024Label-Augmented Dataset Distillationcited by 5
  29. 2025Distilling Dataset into Neural Fieldcited by 5
  30. 2025ImagebindDC: Compressing Multi-modal Data with Imagebind-based Condensationcited by 4
  31. 2023Farzi Data: Autoregressive Data Distillationcited by 3
  32. 2024Distilling Datasets Into Less Than One Imagecited by 3
  33. 2024Dataset Distillation from First Principles: Integrating Core Information Extraction and Purposeful Learningcited by 3
  34. 2025CONCORD: Concept-Informed Diffusion for Dataset Distillationcited by 3
  35. 2025AdaDeDup: Adaptive Hybrid Data Pruning for Efficient Large-Scale Object Detection Trainingcited by 3
  36. 2023Discovering Galaxy Features via Dataset Distillationcited by 2
  37. 2023Towards Model-Based Data Acquisition for Subjective Multi-Task NLP Problemscited by 2
  38. 2024Data Pruning via Separability, Integrity, and Model Uncertainty-Aware Importance Samplingcited by 2

50 most cited of 86 All 86 (JSON)