# Information-Guided Diffusion Sampling for Dataset Distillation

arXiv: https://arxiv.org/abs/2507.04619 · HTML: https://mlsanity.com/paper/2507.04619/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 46 of its 63 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 46)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Flexible Diffusion Modeling of Long Videos](https://mlsanity.com/paper/2205.11495/) (2022)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022)
- [Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory](https://mlsanity.com/paper/2211.10586/) (2022)
- [Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective](https://mlsanity.com/paper/2306.13092/) (2023)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Improved Distribution Matching for Dataset Condensation](https://mlsanity.com/paper/2307.09742/) (2023)
- [Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching](https://mlsanity.com/paper/2310.05773/) (2023)
- [A Comprehensive Survey of Dataset Distillation](https://mlsanity.com/paper/2301.05603/) (2023)
- [On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm](https://mlsanity.com/paper/2312.03526/) (2023)
- [Data Distillation: A Survey](https://mlsanity.com/paper/2301.04272/) (2023)
- [DataDAM: Efficient Dataset Distillation with Attention Matching](https://mlsanity.com/paper/2310.00093/) (2023)
- [Efficient Dataset Distillation via Minimax Diffusion](https://mlsanity.com/paper/2311.15529/) (2023)
- [Synthesizing Informative Training Samples with GAN](https://mlsanity.com/paper/2204.07513/) (2022)
- [D4M: Dataset Distillation via Disentangled Diffusion Model](https://mlsanity.com/paper/2407.15138/) (2024)

## Built on by (most cited first, 10 of 10)

- [Decoupled Audio-Visual Dataset Distillation](https://mlsanity.com/paper/2511.17890/) (2025)
- [ManifoldGD: Training-Free Hierarchical Manifold Guidance for Diffusion-Based Dataset Distillation](https://mlsanity.com/paper/2602.23295/) (2026)
- [FD2: A Dedicated Framework for Fine-Grained Dataset Distillation](https://mlsanity.com/paper/2603.25144/) (2026)
- [Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models](https://mlsanity.com/paper/2605.07194/) (2026)
- [Efficient Video Dataset Distillation via Cluster-Guided Prototype Blending](https://mlsanity.com/paper/2608.03269/) (2026)
- [Difficulty-guided Sampling: Bridging the Target Gap between Dataset Distillation and Downstream Tasks](https://mlsanity.com/paper/2601.10090/) (2026)
- [SAS: Semantic-aware Sampling for Generative Dataset Distillation](https://mlsanity.com/paper/2605.18012/) (2026)
- [Pool-Select-Refine for Allocation-Aware Generative Dataset Distillation](https://mlsanity.com/paper/2606.01920/) (2026)
- [DeCO: Discriminative Evidence Composition for Fine-Grained Dataset Distillation](https://mlsanity.com/paper/2608.25480/) (2026)
- [PSM: Dataset Distillation Based on Precise Statistical Matching by Difficulty](https://mlsanity.com/paper/2609.34299/) (2026)

## Abstract (arXiv)

Dataset distillation aims to create a compact dataset that retains essential information while maintaining model performance. Diffusion models (DMs) have shown promise for this task but struggle in low images-per-class (IPC) settings, where generated samples lack diversity. In this paper, we address this issue from an information-theoretic perspective by identifying two key types of information that a distilled dataset must preserve: (i) prototype information I(X;Y), which captures label-relevant features; and (ii) contextual information H(X | Y), which preserves intra-class variability. Here, (X,Y) represents the pair of random variables corresponding to the input data and its ground truth label, respectively. Observing that the required contextual information scales with IPC, we propose maximizing I(X;Y) + β H(X | Y) during the DM sampling process, where β is IPC-dependent. Since directly computing I(X;Y) and H(X | Y) is intractable, we develop variational estimations to tightly lower-bound these quantities via a data-driven approach. Our approach, information-guided diffusion sampling (IGDS), seamlessly integrates with diffusion models and improves dataset distillation across all IPC settings. Experiments on Tiny ImageNet and ImageNet subsets show that IGDS significantly outperforms existing methods, particularly in low-IPC regimes. The code will be released upon acceptance.