# Dataset Distillation

arXiv: https://arxiv.org/abs/1811.10959 · HTML: https://mlsanity.com/paper/1811.10959/

## Key facts

- Cited by 230 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2018.

## Built on by (most cited first, 30 of 230)

- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Efficient and Modular Implicit Differentiation](https://mlsanity.com/paper/2105.15183/) (2021)
- [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022)
- [DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning](https://mlsanity.com/paper/2204.08499/) (2022)
- [Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation](https://mlsanity.com/paper/2211.11004/) (2022)
- [Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective](https://mlsanity.com/paper/2306.13092/) (2023)
- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [Improved Distribution Matching for Dataset Condensation](https://mlsanity.com/paper/2307.09742/) (2023)
- [A Comprehensive Survey of Dataset Distillation](https://mlsanity.com/paper/2301.05603/) (2023)
- [Dataset Condensation with Contrastive Signals](https://mlsanity.com/paper/2202.02916/) (2022)
- [DC-BENCH: Dataset Condensation Benchmark](https://mlsanity.com/paper/2207.09639/) (2022)
- [Accelerating Dataset Distillation via Model Augmentation](https://mlsanity.com/paper/2212.06152/) (2022)
- [CaT: Balanced Continual Graph Learning with Graph Condensation](https://mlsanity.com/paper/2309.09455/) (2023)
- [Soft-Label Anonymous Gastric X-ray Image Distillation](https://mlsanity.com/paper/2104.02857/) (2021)
- [A Fast Knowledge Distillation Framework for Visual Recognition](https://mlsanity.com/paper/2112.01528/) (2021)
- [Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environments](https://mlsanity.com/paper/2208.11311/) (2022)
- [Efficient Lottery Ticket Finding: Less Data is More](https://mlsanity.com/paper/2106.03225/) (2021)
- [Federated Learning Meets Natural Language Processing: A Survey](https://mlsanity.com/paper/2107.12603/) (2021)
- [An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learning](https://mlsanity.com/paper/2308.00788/) (2023)
- [Delving into Effective Gradient Matching for Dataset Condensation](https://mlsanity.com/paper/2208.00311/) (2022)
- [Elucidating the Design Space of Dataset Condensation](https://mlsanity.com/paper/2404.13733/) (2024)
- [HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks](https://mlsanity.com/paper/2102.02515/) (2021)
- [Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharing](https://mlsanity.com/paper/2209.14635/) (2022)
- [Private Set Generation with Discriminative Information](https://mlsanity.com/paper/2211.04446/) (2022)

## Abstract (arXiv)

Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few gradient descent steps, given a fixed network initialization. We evaluate our method in various initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods.