# Flexible Dataset Distillation: Learn Labels Instead of Images

arXiv: https://arxiv.org/abs/2006.08572 · HTML: https://mlsanity.com/paper/2006.08572/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2020.

## Built on by (most cited first, 30 of 91)

- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory](https://mlsanity.com/paper/2211.10586/) (2022)
- [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022)
- [Graph Condensation for Graph Neural Networks](https://mlsanity.com/paper/2110.07580/) (2021)
- [Dataset Pruning: Reducing Training Data by Examining Generalization Influence](https://mlsanity.com/paper/2205.09329/) (2022)
- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Improved Distribution Matching for Dataset Condensation](https://mlsanity.com/paper/2307.09742/) (2023)
- [Privacy for Free: How does Dataset Condensation Help Privacy?](https://mlsanity.com/paper/2206.00240/) (2022)
- [Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching](https://mlsanity.com/paper/2310.05773/) (2023)
- [Dataset Distillation via Factorization](https://mlsanity.com/paper/2210.16774/) (2022)
- [A Comprehensive Survey of Dataset Distillation](https://mlsanity.com/paper/2301.05603/) (2023)
- [Efficient Dataset Distillation Using Random Feature Approximation](https://mlsanity.com/paper/2210.12067/) (2022)
- [DREAM: Efficient Dataset Distillation by Representative Matching](https://mlsanity.com/paper/2302.14416/) (2023)
- [Data Distillation: A Survey](https://mlsanity.com/paper/2301.04272/) (2023)
- [DataDAM: Efficient Dataset Distillation with Attention Matching](https://mlsanity.com/paper/2310.00093/) (2023)
- [Synthesizing Informative Training Samples with GAN](https://mlsanity.com/paper/2204.07513/) (2022)
- [Accelerating Dataset Distillation via Model Augmentation](https://mlsanity.com/paper/2212.06152/) (2022)
- [DiM: Distilling Dataset into Generative Model](https://mlsanity.com/paper/2303.04707/) (2023)
- [Sequential Subset Matching for Dataset Distillation](https://mlsanity.com/paper/2311.01570/) (2023)
- [HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks](https://mlsanity.com/paper/2102.02515/) (2021)
- [Submodularity In Machine Learning and Artificial Intelligence](https://mlsanity.com/paper/2202.00132/) (2022)
- [Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation](https://mlsanity.com/paper/2404.00563/) (2024)
- [A Label is Worth a Thousand Images in Dataset Distillation](https://mlsanity.com/paper/2406.10485/) (2024)
- [Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error](https://mlsanity.com/paper/2106.09613/) (2021)
- [Label Poisoning is All You Need](https://mlsanity.com/paper/2310.18933/) (2023)
- [Dataset Distillation Using Parameter Pruning](https://mlsanity.com/paper/2209.14609/) (2022)

## Abstract (arXiv)

We study the problem of dataset distillation - creating a small set of synthetic examples capable of training a good model. In particular, we study the problem of label distillation - creating synthetic labels for a small set of real images, and show it to be more effective than the prior image-based approach to dataset distillation. Methodologically, we introduce a more robust and flexible meta-learning algorithm for distillation, as well as an effective first-order strategy based on convex optimization layers. Distilling labels with our new algorithm leads to improved results over prior image-based distillation. More importantly, it leads to clear improvements in flexibility of the distilled dataset in terms of compatibility with off-the-shelf optimizers and diverse neural architectures. Interestingly, label distillation can also be applied across datasets, for example enabling learning Japanese character recognition by training only on synthetically labeled English letters.