# ManifoldGD: Training-Free Hierarchical Manifold Guidance for Diffusion-Based Dataset Distillation

arXiv: https://arxiv.org/abs/2602.23295 · HTML: https://mlsanity.com/paper/2602.23295/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 54 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Builds on (most cited first, 30 of 38)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022)
- [Manifold Preserving Guided Diffusion](https://mlsanity.com/paper/2311.16424/) (2023)
- [CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models](https://mlsanity.com/paper/2406.08070/) (2024)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Dataset Distillation via Factorization](https://mlsanity.com/paper/2210.16774/) (2022)
- [Efficient Dataset Distillation Using Random Feature Approximation](https://mlsanity.com/paper/2210.12067/) (2022)
- [Efficient Dataset Distillation via Minimax Diffusion](https://mlsanity.com/paper/2311.15529/) (2023)
- [D4M: Dataset Distillation via Disentangled Diffusion Model](https://mlsanity.com/paper/2407.15138/) (2024)
- [Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure](https://mlsanity.com/paper/1904.03837/) (2019)
- [Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimization](https://mlsanity.com/paper/2110.06554/) (2021)
- [No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models](https://mlsanity.com/paper/2407.02687/) (2024)
- [Soft-Label Anonymous Gastric X-ray Image Distillation](https://mlsanity.com/paper/2104.02857/) (2021)
- [Tutorial on Diffusion Models for Imaging and Vision](https://mlsanity.com/paper/2403.18103/) (2024)
- [DiM: Distilling Dataset into Generative Model](https://mlsanity.com/paper/2303.04707/) (2023)
- [SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching](https://mlsanity.com/paper/2406.18561/) (2024)
- [Dataset Distillation via Vision-Language Category Prototype](https://mlsanity.com/paper/2506.23580/) (2025)
- [Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios](https://mlsanity.com/paper/2410.17193/) (2024)
- [Dataset Condensation via Generative Model](https://mlsanity.com/paper/2309.07698/) (2023)
- [The Data Addition Dilemma](https://mlsanity.com/paper/2408.04154/) (2024)
- [SecDD: Efficient and Secure Method for Remotely Training Neural Networks](https://mlsanity.com/paper/2009.09155/) (2020)
- [Neural Spectral Decomposition for Dataset Distillation](https://mlsanity.com/paper/2408.16236/) (2024)

## Built on by (most cited first, 4 of 4)

- [Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?](https://mlsanity.com/paper/2606.18209/) (2026)
- [MGPO: Manifold-Guided Diffusion Alignment for Task-Aware Dataset Distillation](https://mlsanity.com/paper/2610.05252/) (2026)
- [IntactWorld: Joint World Modeling with Intact Features](https://mlsanity.com/paper/2610.11174/) (2026)
- [MAGNET: Manifold-Aware Graph Diffusion Network for Connectome Generation](https://mlsanity.com/paper/2610.17014/) (2026)

## Abstract (arXiv)

In recent times, large datasets hinder efficient model training while also containing redundant concepts. Dataset distillation aims to synthesize compact datasets that preserve the knowledge of large-scale training sets while drastically reducing storage and computation. Recent advances in diffusion models have enabled training-free distillation by leveraging pre-trained generative priors; however, existing guidance strategies remain limited. Current score-based methods either perform unguided denoising or rely on simple mode-based guidance toward instance prototype centroids (IPC centroids), which often are rudimentary and suboptimal. We propose Manifold-Guided Distillation (ManifoldGD), a training-free diffusion-based framework that integrates manifold consistent guidance at every denoising timestep. Our method employs IPCs computed via a hierarchical, divisive clustering of VAE latent features, yielding a multi-scale coreset of IPCs that captures both coarse semantic modes and fine intra-class variability. Using a local neighborhood of the extracted IPC centroids, we create the latent manifold for each diffusion denoising timestep. At each denoising step, we project the mode-alignment vector onto the local tangent space of the estimated latent manifold, thus constraining the generation trajectory to remain manifold-faithful while preserving semantic consistency. This formulation improves representativeness, diversity, and image fidelity without requiring any model retraining. Empirical results demonstrate consistent gains over existing training-free and training-based baselines in terms of FID, l2 distance among real and synthetic dataset embeddings, and classification accuracy, establishing ManifoldGD as the first geometry-aware training-free data distillation framework.