# What is Dataset Distillation Learning?

arXiv: https://arxiv.org/abs/2406.04284 · HTML: https://mlsanity.com/paper/2406.04284/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022)
- [Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory](https://mlsanity.com/paper/2211.10586/) (2022)
- [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022)
- [DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning](https://mlsanity.com/paper/2204.08499/) (2022)
- [Improved Distribution Matching for Dataset Condensation](https://mlsanity.com/paper/2307.09742/) (2023)
- [Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks](https://mlsanity.com/paper/2206.02916/) (2022)
- [Delving into Effective Gradient Matching for Dataset Condensation](https://mlsanity.com/paper/2208.00311/) (2022)
- [On Implicit Bias in Overparameterized Bilevel Optimization](https://mlsanity.com/paper/2212.14032/) (2022)
- [Frequency Domain-based Dataset Distillation](https://mlsanity.com/paper/2311.08819/) (2023)
- [Error Discovery by Clustering Influence Embeddings](https://mlsanity.com/paper/2312.04712/) (2023)
- [When less is more: Simplifying inputs aids neural network understanding](https://mlsanity.com/paper/2201.05610/) (2022)
- [On the Size and Approximation Error of Distilled Sets](https://mlsanity.com/paper/2305.14113/) (2023)

## Built on by (most cited first, 13 of 13)

- [A Label is Worth a Thousand Images in Dataset Distillation](https://mlsanity.com/paper/2406.10485/) (2024)
- [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025)
- [Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios](https://mlsanity.com/paper/2410.17193/) (2024)
- [Dataset distillation for memorized data: Soft labels can leak held-out teacher knowledge](https://mlsanity.com/paper/2506.14457/) (2025)
- [Data Diversity as Implicit Regularization: How Does Diversity Shape the Weight Space of Deep Neural Networks?](https://mlsanity.com/paper/2410.14602/) (2024)
- [Going Beyond Feature Similarity: Effective Dataset Distillation based on Class-Aware Conditional Mutual Information](https://mlsanity.com/paper/2412.09945/) (2024)
- [Diffusion Models as Dataset Distillation Priors](https://mlsanity.com/paper/2510.17421/) (2025)
- [GVD: Guiding Video Diffusion Model for Scalable Video Distillation](https://mlsanity.com/paper/2507.22360/) (2025)
- [DiRe: Diversity-promoting Regularization for Dataset Condensation](https://mlsanity.com/paper/2512.13083/) (2025)
- [Structural Assessment for Understanding and Guiding Dataset Distillation in Discrete Token Space](https://mlsanity.com/paper/2606.21705/) (2026)
- [Distill Video Datasets into Images](https://mlsanity.com/paper/2512.14621/) (2025)
- [CD^2: Constrained Dataset Distillation for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2601.08519/) (2026)
- [An Approximation Algorithm for Graph Label Selection](https://mlsanity.com/paper/2605.18623/) (2026)

## Abstract (arXiv)

Dataset distillation has emerged as a strategy to overcome the hurdles associated with large datasets by learning a compact set of synthetic data that retains essential information from the original dataset. While distilled data can be used to train high performing models, little is understood about how the information is stored. In this study, we posit and answer three questions about the behavior, representativeness, and point-wise information content of distilled data. We reveal distilled data cannot serve as a substitute for real data during training outside the standard evaluation setting for dataset distillation. Additionally, the distillation process retains high task performance by compressing information related to the early training dynamics of real models. Finally, we provide an framework for interpreting distilled data and reveal that individual distilled data points contain meaningful semantic information. This investigation sheds light on the intricate nature of distilled data, providing a better understanding on how they can be effectively utilized.