# Generalizing Dataset Distillation via Deep Generative Prior

arXiv: https://arxiv.org/abs/2305.01649 · HTML: https://mlsanity.com/paper/2305.01649/

## Key facts

- Cited by 126 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 82 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 30 of 64)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://mlsanity.com/paper/2202.00273/) (2022)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Pivotal Tuning for Latent-based Editing of Real Images](https://mlsanity.com/paper/2106.05744/) (2021)
- [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)

## Built on by (most cited first, 30 of 126)

- [On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm](https://mlsanity.com/paper/2312.03526/) (2023)
- [Efficient Dataset Distillation via Minimax Diffusion](https://mlsanity.com/paper/2311.15529/) (2023)
- [D4M: Dataset Distillation via Disentangled Diffusion Model](https://mlsanity.com/paper/2407.15138/) (2024)
- [CaT: Balanced Continual Graph Learning with Graph Condensation](https://mlsanity.com/paper/2309.09455/) (2023)
- [M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy](https://mlsanity.com/paper/2312.15927/) (2023)
- [Does Graph Distillation See Like Vision Dataset Counterpart?](https://mlsanity.com/paper/2310.09192/) (2023)
- [Sequential Subset Matching for Dataset Distillation](https://mlsanity.com/paper/2311.01570/) (2023)
- [Dataset Distillation via the Wasserstein Metric](https://mlsanity.com/paper/2311.18531/) (2023)
- [A Label is Worth a Thousand Images in Dataset Distillation](https://mlsanity.com/paper/2406.10485/) (2024)
- [Frequency Domain-based Dataset Distillation](https://mlsanity.com/paper/2311.08819/) (2023)
- [A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances](https://mlsanity.com/paper/2505.17799/) (2025)
- [Dataset Distillation via Curriculum Data Synthesis in Large Data Era](https://mlsanity.com/paper/2311.18838/) (2023)
- [Dataset Distillation by Automatic Training Trajectories](https://mlsanity.com/paper/2407.14245/) (2024)
- [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025)
- [Unlocking Dataset Distillation with Diffusion Models](https://mlsanity.com/paper/2403.03881/) (2024)
- [DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation](https://mlsanity.com/paper/2404.00264/) (2024)
- [Multisize Dataset Condensation](https://mlsanity.com/paper/2403.06075/) (2024)
- [Dataset Distillation via Vision-Language Category Prototype](https://mlsanity.com/paper/2506.23580/) (2025)
- [Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios](https://mlsanity.com/paper/2410.17193/) (2024)
- [Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality](https://mlsanity.com/paper/2310.06982/) (2023)
- [Low-Rank Similarity Mining for Multimodal Dataset Distillation](https://mlsanity.com/paper/2406.03793/) (2024)
- [Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents](https://mlsanity.com/paper/2312.01537/) (2023)
- [DANCE: Dual-View Distribution Alignment for Dataset Condensation](https://mlsanity.com/paper/2406.01063/) (2024)
- [Neural Spectral Decomposition for Dataset Distillation](https://mlsanity.com/paper/2408.16236/) (2024)
- [CaO2: Rectifying Inconsistencies in Diffusion-Based Dataset Distillation](https://mlsanity.com/paper/2506.22637/) (2025)
- [Hierarchical Features Matter: A Deep Exploration of Progressive Parameterization Method for Dataset Distillation](https://mlsanity.com/paper/2406.05704/) (2024)
- [Generative Dataset Distillation Based on Diffusion Model](https://mlsanity.com/paper/2408.08610/) (2024)
- [Can pre-trained models assist in dataset distillation?](https://mlsanity.com/paper/2310.03295/) (2023)
- [Group Distributionally Robust Dataset Distillation with Risk Minimization](https://mlsanity.com/paper/2402.04676/) (2024)
- [DD-Ranking: Rethinking the Evaluation of Dataset Distillation](https://mlsanity.com/paper/2505.13300/) (2025)

## Abstract (arXiv)

Dataset Distillation aims to distill an entire dataset's knowledge into a few synthetic images. The idea is to synthesize a small number of synthetic data points that, when given to a learning algorithm as training data, result in a model approximating one trained on the original data. Despite recent progress in the field, existing dataset distillation methods fail to generalize to new architectures and scale to high-resolution datasets. To overcome the above issues, we propose to use the learned prior from pre-trained deep generative models to synthesize the distilled data. To achieve this, we present a new optimization algorithm that distills a large number of images into a few intermediate feature vectors in the generative model's latent space. Our method augments existing techniques, significantly improving cross-architecture generalization in all settings.