# Unlocking Dataset Distillation with Diffusion Models

arXiv: https://arxiv.org/abs/2403.03881 · HTML: https://mlsanity.com/paper/2403.03881/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 6 Mar 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://mlsanity.com/paper/2202.00273/) (2022)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [GAN Inversion: A Survey](https://mlsanity.com/paper/2101.05278/) (2021)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Predictability and Surprise in Large Generative Models](https://mlsanity.com/paper/2202.07785/) (2022)
- [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022)
- [Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective](https://mlsanity.com/paper/2306.13092/) (2023)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Privacy for Free: How does Dataset Condensation Help Privacy?](https://mlsanity.com/paper/2206.00240/) (2022)
- [Neural Photo Editing with Introspective Adversarial Networks](https://mlsanity.com/paper/1609.07093/) (2016)
- [Efficient Dataset Distillation via Minimax Diffusion](https://mlsanity.com/paper/2311.15529/) (2023)
- [Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching](https://mlsanity.com/paper/2311.17950/) (2023)
- [Diffusion Models, Image Super-Resolution And Everything: A Survey](https://mlsanity.com/paper/2401.00736/) (2024)
- [Discovering Failure Modes of Text-guided Diffusion Models via Adversarial Search](https://mlsanity.com/paper/2306.00974/) (2023)
- [Less is More: Proxy Datasets in NAS approaches](https://mlsanity.com/paper/2203.06905/) (2022)

## Built on by (most cited first, 23 of 23)

- [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025)
- [Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios](https://mlsanity.com/paper/2410.17193/) (2024)
- [Generative Dataset Distillation Based on Diffusion Model](https://mlsanity.com/paper/2408.08610/) (2024)
- [DD-Ranking: Rethinking the Evaluation of Dataset Distillation](https://mlsanity.com/paper/2505.13300/) (2025)
- [Diversity-Driven Generative Dataset Distillation Based on Diffusion Model with Self-Adaptive Memory](https://mlsanity.com/paper/2505.19469/) (2025)
- [MGD3: Mode-Guided Dataset Distillation using Diffusion Models](https://mlsanity.com/paper/2505.18963/) (2025)
- [A Study in Dataset Pruning for Image Super-Resolution](https://mlsanity.com/paper/2403.17083/) (2024)
- [Distill the Best, Ignore the Rest: Improving Dataset Distillation with Loss-Value-Based Pruning](https://mlsanity.com/paper/2411.12115/) (2024)
- [Distilling Dataset into Neural Field](https://mlsanity.com/paper/2503.04835/) (2025)
- [Prioritize Alignment in Dataset Distillation](https://mlsanity.com/paper/2408.03360/) (2024)
- [DMGD: Train-Free Dataset Distillation with Semantic-Distribution Matching in Diffusion Models](https://mlsanity.com/paper/2605.03877/) (2026)
- [Webcam-based Pupil Diameter Prediction Benefits from Upscaling](https://mlsanity.com/paper/2408.10397/) (2024)
- [Dataset Distillation as Pushforward Optimal Quantization](https://mlsanity.com/paper/2501.07681/) (2025)
- [Latent Video Dataset Distillation](https://mlsanity.com/paper/2504.17132/) (2025)
- [Geometry-Aware Dataset Condensation for Diffusion Model Training](https://mlsanity.com/paper/2606.05883/) (2026)
- [UDD: Dataset Distillation via Mining Underutilized Regions](https://mlsanity.com/paper/2408.16268/) (2024)
- [Just Leaf It: Accelerating Diffusion Classifiers with Hierarchical Class Pruning](https://mlsanity.com/paper/2411.12073/) (2024)
- [A Dataset is Worth 1 MB](https://mlsanity.com/paper/2602.23358/) (2026)
- [IMS3: Breaking Distributional Aggregation in Diffusion-Based Dataset Distillation](https://mlsanity.com/paper/2603.13960/) (2026)
- [Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment](https://mlsanity.com/paper/2603.27987/) (2026)
- [D3S2: Diffusion-Guided Dataset Distillation for Semantic Segmentation](https://mlsanity.com/paper/2605.25022/) (2026)
- [Pool-Select-Refine for Allocation-Aware Generative Dataset Distillation](https://mlsanity.com/paper/2606.01920/) (2026)
- [ProMoDD: Dataset Distillation for 3D Human Motion via Prototype-Conditioned Diffusion Models](https://mlsanity.com/paper/2610.16134/) (2026)

## Abstract (arXiv)

Dataset distillation seeks to condense datasets into smaller but highly representative synthetic samples. While diffusion models now lead all generative benchmarks, current distillation methods avoid them and rely instead on GANs or autoencoders, or, at best, sampling from a fixed diffusion prior. This trend arises because naive backpropagation through the long denoising chain leads to vanishing gradients, which prevents effective synthetic sample optimization. To address this limitation, we introduce Latent Dataset Distillation with Diffusion Models (LD3M), the first method to learn gradient-based distilled latents and class embeddings end-to-end through a pre-trained latent diffusion model. A linearly decaying skip connection, injected from the initial noisy state into every reverse step, preserves the gradient signal across dozens of timesteps without requiring diffusion weight fine-tuning. Across multiple ImageNet subsets at 128x128 and 256x256, LD3M improves downstream accuracy by up to 4.8 percentage points (1 IPC) and 4.2 points (10 IPC) over the prior state-of-the-art. The code for LD3M is provided at https://github.com/Brian-Moser/prune_and_distill.