mlsanityData to 7 Oct 2026

Dataset Distillation: A Comprehensive Review

arXiv 2301.07014 ↗First posted 17 Jan 2023cs.LGcited by 126

Not on the map. Why?

Abstract (arXiv)

Recent success of deep learning is largely attributed to the sheer amount of data used for training deep neural networks.Despite the unprecedented success, the massive data, unfortunately, significantly increases the burden on storage and transmission and further gives rise to a cumbersome model training process. Besides, relying on the raw data for training per se yields concerns about privacy and copyright. To alleviate these shortcomings, dataset distillation~(DD), also known as dataset condensation (DC), was introduced and has recently attracted much research attention in the community. Given an original dataset, DD aims to derive a much smaller dataset containing synthetic samples, based on which the trained models yield performance comparable with those trained on the original dataset. In this paper, we give a comprehensive review and summary of recent advances in DD and its application. We first introduce the task formally and propose an overall algorithmic framework followed by all existing DD methods. Next, we provide a systematic taxonomy of current methodologies in this area, and discuss their theoretical interconnections. We also present current challenges in DD through extensive experiments and envision possible directions for future works.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Builds on 136 of its 188 references that are in the map. First posted on arXiv 17 Jan 2023.

Builds on

136 of 188 references.

References · 136

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2017Graph Attention Networkscited by 8,854
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  11. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
38 more
  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  11. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  12. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  13. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  14. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  15. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  16. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  17. 2014Conditional Generative Adversarial Netscited by 2,152
  18. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  19. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  20. 2020Knowledge Distillation: A Surveycited by 1,578
  21. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  22. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  23. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  24. 2019Relational Knowledge Distillationcited by 873
  25. 2018Hierarchical Graph Representation Learning with Differentiable Poolingcited by 786
  26. 2019Contrastive Representation Distillationcited by 734
  27. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  28. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  29. 2019Graph Neural Networks for Social Recommendationcited by 711
  30. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  31. 2017Deep Neural Networks as Gaussian Processescited by 617
  32. 2018Rethinking the Value of Network Pruningcited by 595
  33. 2020What is the State of Neural Network Pruning?cited by 578
  34. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  35. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  36. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  37. 2020Dataset Condensation with Gradient Matchingcited by 510
  38. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 509

50 most cited of 136 All 136 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 31; 2024: 38; 2025: 37; 2026: 20.

126 papers.

Sort:

By area:

  1. 2023DepGraph: Towards Any Structural Pruningcited by 202
  2. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  3. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  4. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 107
  5. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 76
  6. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  7. 2024UniTraj: A Unified Framework for Scalable Vehicle Trajectory Predictioncited by 51
  8. 2023Sequential Subset Matching for Dataset Distillationcited by 40
  9. 2023Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruningcited by 38
  10. 2023Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learningcited by 26
  11. 2023Too Large; Data Reduction for Vision-Language Pre-Trainingcited by 26
  12. 2025The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutionscited by 24
38 more
  1. 2024Graph Condensation: A Surveycited by 23
  2. 2024Rethinking and Accelerating Graph Condensation: A Training-Free Approach with Class Partitioncited by 23
  3. 2023Deep Graph Reprogrammingcited by 22
  4. 2023SlimSAM: 0.1% Data Makes Segment Anything Slimcited by 20
  5. 2024DANCE: Dual-View Distribution Alignment for Dataset Condensationcited by 15
  6. 2023Towards Trustworthy Dataset Distillationcited by 14
  7. 2024Automated data processing and feature engineering for deep learning and big data applications: a surveycited by 14
  8. 2023Rethinking Data Distillation: Do Not Overlook Calibrationcited by 13
  9. 2024Self-supervised Dataset Distillation: A Good Compression Is All You Needcited by 13
  10. 2024Dataset Quantization with Active Learning based Adaptive Samplingcited by 13
  11. 2024Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensatorcited by 13
  12. 2024Teddy: Efficient Large-Scale Dataset Distillation via Taylor-Approximated Matchingcited by 13
  13. 2025TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendationcited by 13
  14. 2025CaO2: Rectifying Inconsistencies in Diffusion-Based Dataset Distillationcited by 13
  15. 2023Improving Expressivity of GNNs with Subgraph-specific Factor Embedded Normalizationcited by 12
  16. 2023Efficiency is Not Enough: A Critical Perspective of Environmentally Sustainable AIcited by 12
  17. 2023Can pre-trained models assist in dataset distillation?cited by 12
  18. 2024One-shot Federated Learning via Synthetic Distiller-Distillate Communicationcited by 12
  19. 2024Distilling the Knowledge in Data Pruningcited by 11
  20. 2024Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimizationcited by 11
  21. 2023Rethinking Backdoor Attacks on Dataset Distillation: A Kernel Method Perspectivecited by 10
  22. 2023Master: Meta Style Transformer for Controllable Zero-Shot and Few-Shot Artistic Style Transfercited by 9
  23. 2024A Survey on Graph Condensationcited by 9
  24. 2024Ungeneralizable Examplescited by 9
  25. 2024Zero-Shot Coreset Selection via Iterative Subspace Samplingcited by 9
  26. 2025Dataset distillation for memorized data: Soft labels can leak held-out teacher knowledgecited by 9
  27. 2023Fair Wasserstein Coresetscited by 8
  28. 2024Anytime Continual Learning for Open Vocabulary Classificationcited by 8
  29. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  30. 2023Dataset Distillation Meets Provable Subset Selectioncited by 7
  31. 2024GC4NC: A Benchmark Framework for Graph Condensation on Node Classification with New Insightscited by 7
  32. 2023Federated Learning on Virtual Heterogeneous Data with Local-global Distillationcited by 6
  33. 2023DREAM+: Efficient Dataset Distillation by Bidirectional Representative Matchingcited by 6
  34. 2023Data Optimization in Deep Learning: A Surveycited by 6
  35. 2024Progressive trajectory matching for medical dataset distillationcited by 5
  36. 2024Dataset Distillation for Offline Reinforcement Learningcited by 5
  37. 2025Distilling Dataset into Neural Fieldcited by 5
  38. 2025Dataset Condensation with Color Compensationcited by 5

50 most cited of 126 All 126 (JSON)