mlsanityData to 14 Oct 2026

A Comprehensive Survey of Dataset Distillation

arXiv 2301.05603 ↗First posted 13 Jan 2023cs.LGcited by 115

Not on the map. Why?

Abstract (arXiv)

Deep learning technology has developed unprecedentedly in the last decade and has become the primary choice in many application domains. This progress is mainly attributed to a systematic collaboration in which rapidly growing computing resources encourage advanced algorithms to deal with massive data. However, it has gradually become challenging to handle the unlimited growth of data with limited computing power. To this end, diverse approaches are proposed to improve data processing efficiency. Dataset distillation, a dataset reduction method, addresses this problem by synthesizing a small typical dataset from substantial data and has attracted much attention from the deep learning community. Existing dataset distillation methods can be taxonomized into meta-learning and data matching frameworks according to whether they explicitly mimic the performance of target data. Although dataset distillation has shown surprising performance in compressing datasets, there are still several limitations such as distilling high-resolution data or data with complex label spaces. This paper provides a holistic understanding of dataset distillation from multiple aspects, including distillation frameworks and algorithms, factorized dataset distillation, performance comparison, and applications. Finally, we discuss challenges and promising directions to further promote future studies on dataset distillation.

Key facts. Cited by 115 papers in the map (data through 14 Oct 2026). Builds on 108 of its 158 references that are in the map. First posted on arXiv 13 Jan 2023.

Builds on

108 of 158 references.

References · 108

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  12. 2016Densely Connected Convolutional Networkscited by 6,720
38 more
  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2016Wide Residual Networkscited by 3,070
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  9. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  10. 2020Knowledge Distillation: A Surveycited by 1,603
  11. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  13. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  14. 2017Mixed Precision Trainingcited by 1,156
  15. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  16. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  17. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  18. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  19. 2020Image Segmentation Using Deep Learning: A Surveycited by 660
  20. 2018End-to-End Incremental Learningcited by 597
  21. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  22. 2020Dataset Condensation with Gradient Matchingcited by 516
  23. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 498
  24. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  25. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  26. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
  27. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 403
  28. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  29. 2015Deep Learning with Limited Numerical Precisioncited by 376
  30. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 370
  31. 2021Dataset Condensation with Distribution Matchingcited by 365
  32. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  33. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  34. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 321
  35. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  36. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 291
  37. 2018Dataset Distillationcited by 230
  38. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230

50 most cited of 108 All 108 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 33; 2025: 37; 2026: 32.

115 papers.

Sort:

By area:

  1. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 77
  2. 2023Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Datacited by 71
  3. 2023Sequential Subset Matching for Dataset Distillationcited by 40
  4. 2024Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustmentcited by 33
  5. 2023Dataset Distillation via the Wasserstein Metriccited by 31
  6. 2024Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defensescited by 30
  7. 2025The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutionscited by 26
  8. 2024Graph Condensation: A Surveycited by 23
  9. 2024Rethinking and Accelerating Graph Condensation: A Training-Free Approach with Class Partitioncited by 23
  10. 2024Large Language Model Agent as a Mechanical Designercited by 21
  11. 2024A CLIP-Powered Framework for Robust and Generalizable Data Selectioncited by 21
  12. 2025Dataset Distillation via Vision-Language Category Prototypecited by 19
38 more
  1. 2024Importance-Aware Adaptive Dataset Distillationcited by 18
  2. 2024Generative Dataset Distillation: Balancing Global Structure and Local Detailscited by 18
  3. 2023Graph Distillation with Eigenbasis Matchingcited by 16
  4. 2025Hyperbolic Dataset Distillationcited by 15
  5. 2024Self-supervised Dataset Distillation: A Good Compression Is All You Needcited by 14
  6. 2024Federated Learning Priorities Under the European Union Artificial Intelligence Actcited by 13
  7. 2024Hierarchical Features Matter: A Deep Exploration of Progressive Parameterization Method for Dataset Distillationcited by 13
  8. 2024Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensatorcited by 13
  9. 2024Generative Dataset Distillation Based on Diffusion Modelcited by 13
  10. 2025TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendationcited by 13
  11. 2023Can pre-trained models assist in dataset distillation?cited by 12
  12. 2025Diversity-Driven Generative Dataset Distillation Based on Diffusion Model with Self-Adaptive Memorycited by 10
  13. 2025Information-Guided Diffusion Sampling for Dataset Distillationcited by 10
  14. 2024Adaptive Dataset Quantizationcited by 9
  15. 2023SeiT: Storage-Efficient Vision Training with Tokens Using 1% of Pixel Storagecited by 8
  16. 2023Fair Wasserstein Coresetscited by 8
  17. 2023Dataset Distillation in Latent Spacecited by 8
  18. 2024RobGC: Towards Robust Graph Condensationcited by 8
  19. 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 8
  20. 2023Query Understanding in the Age of Large Language Modelscited by 7
  21. 2023DREAM+: Efficient Dataset Distillation by Bidirectional Representative Matchingcited by 6
  22. 2023Data Optimization in Deep Learning: A Surveycited by 6
  23. 2024Contrastive Graph Condensation: Advancing Data Versatility through Self-Supervised Learningcited by 6
  24. 2025When Dynamic Data Selection Meets Data Augmentationcited by 6
  25. 2023Embarassingly Simple Dataset Distillationcited by 5
  26. 2024Progressive trajectory matching for medical dataset distillationcited by 5
  27. 2024Behaviour Distillationcited by 5
  28. 2024Dataset Distillation for Offline Reinforcement Learningcited by 5
  29. 2025Distilling Dataset into Neural Fieldcited by 5
  30. 2025Dataset Condensation with Color Compensationcited by 5
  31. 2025Beyond Random: Automatic Inner-loop Optimization in Dataset Distillationcited by 5
  32. 2024Dataset Distillation for Histopathology Image Classificationcited by 4
  33. 2024Towards Consistent and Efficient Dataset Distillation via Diffusion-Driven Selectioncited by 4
  34. 2025Simple yet Effective Graph Distillation via Clusteringcited by 4
  35. 2026LAPS: A Length-Aware-Prefill LLM Serving Systemcited by 4
  36. 2026EVLF: Early Vision-Language Fusion for Generative Dataset Distillationcited by 4
  37. 2024Dataset Distillation from First Principles: Integrating Core Information Extraction and Purposeful Learningcited by 3
  38. 2024Dataset Distillation-based Hybrid Federated Learning on Non-IID Datacited by 3

50 most cited of 115 All 115 (JSON)