# The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions

arXiv: https://arxiv.org/abs/2502.05673 · HTML: https://mlsanity.com/paper/2502.05673/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 154 of its 205 references that are in the map.
- First posted on arXiv 8 Feb 2025.

## Builds on (most cited first, 30 of 154)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://mlsanity.com/paper/2202.00273/) (2022)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Backdoor Learning: A Survey](https://mlsanity.com/paper/2007.08745/) (2020)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Flow Matching on General Geometries](https://mlsanity.com/paper/2302.03660/) (2023)

## Built on by (most cited first, 26 of 26)

- [A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances](https://mlsanity.com/paper/2505.17799/) (2025)
- [Dataset Distillation via Vision-Language Category Prototype](https://mlsanity.com/paper/2506.23580/) (2025)
- [Hyperbolic Dataset Distillation](https://mlsanity.com/paper/2505.24623/) (2025)
- [FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation](https://mlsanity.com/paper/2506.24125/) (2025)
- [Task-Specific Generative Dataset Distillation with Difficulty-Guided Sampling](https://mlsanity.com/paper/2507.03331/) (2025)
- [Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation](https://mlsanity.com/paper/2512.00308/) (2025)
- [EVLF: Early Vision-Language Fusion for Generative Dataset Distillation](https://mlsanity.com/paper/2603.07476/) (2026)
- [Diffusion Models as Dataset Distillation Priors](https://mlsanity.com/paper/2510.17421/) (2025)
- [Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling](https://mlsanity.com/paper/2511.18858/) (2025)
- [Multimodal Distribution Matching for Vision-Language Dataset Distillation](https://mlsanity.com/paper/2605.23482/) (2026)
- [Dataset Distillation as Data Compression: A Rate-Utility Perspective](https://mlsanity.com/paper/2507.17221/) (2025)
- [A Discrepancy-Based Perspective on Dataset Condensation](https://mlsanity.com/paper/2509.10367/) (2025)
- [Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift](https://mlsanity.com/paper/2512.15647/) (2025)
- [Geometry-Aware Dataset Condensation for Diffusion Model Training](https://mlsanity.com/paper/2606.05883/) (2026)
- [Efficient Video Dataset Distillation via Cluster-Guided Prototype Blending](https://mlsanity.com/paper/2608.03269/) (2026)
- [Distill Video Datasets into Images](https://mlsanity.com/paper/2512.14621/) (2025)
- [Foreground-Aware Dataset Distillation via Dynamic Patch Selection](https://mlsanity.com/paper/2601.02727/) (2026)
- [Difficulty-guided Sampling: Bridging the Target Gap between Dataset Distillation and Downstream Tasks](https://mlsanity.com/paper/2601.10090/) (2026)
- [Omnimodal Dataset Distillation via High-order Proxy Alignment](https://mlsanity.com/paper/2604.10666/) (2026)
- [SAS: Semantic-aware Sampling for Generative Dataset Distillation](https://mlsanity.com/paper/2605.18012/) (2026)
- [Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?](https://mlsanity.com/paper/2606.18209/) (2026)
- [Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation](https://mlsanity.com/paper/2606.29464/) (2026)
- [Dataset Distillation Based on Saliency-Driven Prototype Alignment](https://mlsanity.com/paper/2607.25318/) (2026)
- [Self-Supervised Representation-Guided Generative Dataset Distillation](https://mlsanity.com/paper/2608.03218/) (2026)
- [PSM: Dataset Distillation Based on Precise Statistical Matching by Difficulty](https://mlsanity.com/paper/2609.34299/) (2026)
- [X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation](https://mlsanity.com/paper/2610.13687/) (2026)

## Abstract (arXiv)

Dataset distillation, which condenses large-scale datasets into compact synthetic representations, has emerged as a critical solution for training modern deep learning models efficiently. While prior surveys focus on developments before 2023, this work comprehensively reviews recent advances, emphasizing scalability to large-scale datasets such as ImageNet-1K and ImageNet-21K. We categorize progress into a few key methodologies: trajectory matching, gradient matching, distribution matching, scalable generative approaches, and decoupling optimization mechanisms. As a comprehensive examination of recent dataset distillation advances, this survey highlights breakthrough innovations: the SRe2L framework for efficient and effective condensation, soft label strategies that significantly enhance model accuracy, and lossless distillation techniques that maximize compression while maintaining performance. Beyond these methodological advancements, we address critical challenges, including robustness against adversarial and backdoor attacks, effective handling of non-IID data distributions. Additionally, we explore emerging applications in video and audio processing, multi-modal learning, medical imaging, and scientific computing, highlighting its domain versatility. By offering extensive performance comparisons and actionable research directions, this survey equips researchers and practitioners with practical insights to advance efficient and generalizable dataset distillation, paving the way for future innovations.