# Dataset Distillation via Adversarial Prediction Matching

arXiv: https://arxiv.org/abs/2312.08912 · HTML: https://mlsanity.com/paper/2312.08912/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022)
- [Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory](https://mlsanity.com/paper/2211.10586/) (2022)
- [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022)
- [Auto-Keras: An Efficient Neural Architecture Search System](https://mlsanity.com/paper/1806.10282/) (2018)
- [Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation](https://mlsanity.com/paper/2211.11004/) (2022)
- [Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective](https://mlsanity.com/paper/2306.13092/) (2023)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Flexible Dataset Distillation: Learn Labels Instead of Images](https://mlsanity.com/paper/2006.08572/) (2020)
- [Data-Free Adversarial Distillation](https://mlsanity.com/paper/1912.11006/) (2019)
- [Meta-Learning with Adaptive Hyperparameters](https://mlsanity.com/paper/2011.00209/) (2020)
- [DC-BENCH: Dataset Condensation Benchmark](https://mlsanity.com/paper/2207.09639/) (2022)
- [Role-Wise Data Augmentation for Knowledge Distillation](https://mlsanity.com/paper/2004.08861/) (2020)

## Built on by (most cited first, 9 of 9)

- [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025)
- [Self-supervised Dataset Distillation: A Good Compression Is All You Need](https://mlsanity.com/paper/2404.07976/) (2024)
- [Dataset Distillation via Committee Voting](https://mlsanity.com/paper/2501.07575/) (2025)
- [FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation](https://mlsanity.com/paper/2506.24125/) (2025)
- [EVLF: Early Vision-Language Fusion for Generative Dataset Distillation](https://mlsanity.com/paper/2603.07476/) (2026)
- [Beyond Size and Class Balance: Alpha as a New Dataset Quality Metric for Deep Learning](https://mlsanity.com/paper/2407.15724/) (2024)
- [FD2: A Dedicated Framework for Fine-Grained Dataset Distillation](https://mlsanity.com/paper/2603.25144/) (2026)
- [Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift](https://mlsanity.com/paper/2512.15647/) (2025)
- [PSM: Dataset Distillation Based on Precise Statistical Matching by Difficulty](https://mlsanity.com/paper/2609.34299/) (2026)

## Abstract (arXiv)

Dataset distillation is the technique of synthesizing smaller condensed datasets from large original datasets while retaining necessary information to persist the effect. In this paper, we approach the dataset distillation problem from a novel perspective: we regard minimizing the prediction discrepancy on the real data distribution between models, which are respectively trained on the large original dataset and on the small distilled dataset, as a conduit for condensing information from the raw data into the distilled version. An adversarial framework is proposed to solve the problem efficiently. In contrast to existing distillation methods involving nested optimization or long-range gradient unrolling, our approach hinges on single-level optimization. This ensures the memory efficiency of our method and provides a flexible tradeoff between time and memory budgets, allowing us to distil ImageNet-1K using a minimum of only 6.5GB of GPU memory. Under the optimal tradeoff strategy, it requires only 2.5× less memory and 5× less runtime compared to the state-of-the-art. Empirically, our method can produce synthetic datasets just 10% the size of the original, yet achieve, on average, 94% of the test accuracy of models trained on the full original datasets including ImageNet-1K, significantly surpassing state-of-the-art. Additionally, extensive tests reveal that our distilled datasets excel in cross-architecture generalization capabilities.