# Generative Dataset Distillation Based on Self-knowledge Distillation

arXiv: https://arxiv.org/abs/2501.04202 · HTML: https://mlsanity.com/paper/2501.04202/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 8 Jan 2025.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022)
- [Fair Clustering Through Fairlets](https://mlsanity.com/paper/1802.05733/) (2018)
- [Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data](https://mlsanity.com/paper/1912.07768/) (2019)
- [Soft-Label Anonymous Gastric X-ray Image Distillation](https://mlsanity.com/paper/2104.02857/) (2021)
- [Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environments](https://mlsanity.com/paper/2208.11311/) (2022)
- [DiM: Distilling Dataset into Generative Model](https://mlsanity.com/paper/2303.04707/) (2023)
- [Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharing](https://mlsanity.com/paper/2209.14635/) (2022)
- [Dataset Distillation for Medical Dataset Sharing](https://mlsanity.com/paper/2209.14603/) (2022)
- [Importance-Aware Adaptive Dataset Distillation](https://mlsanity.com/paper/2401.15863/) (2024)
- [Generative Dataset Distillation: Balancing Global Structure and Local Details](https://mlsanity.com/paper/2404.17732/) (2024)
- [Generative Dataset Distillation Based on Diffusion Model](https://mlsanity.com/paper/2408.08610/) (2024)

## Built on by (most cited first, 5 of 5)

- [Hyperbolic Dataset Distillation](https://mlsanity.com/paper/2505.24623/) (2025)
- [Information-Guided Diffusion Sampling for Dataset Distillation](https://mlsanity.com/paper/2507.04619/) (2025)
- [Task-Specific Generative Dataset Distillation with Difficulty-Guided Sampling](https://mlsanity.com/paper/2507.03331/) (2025)
- [Foreground-Aware Dataset Distillation via Dynamic Patch Selection](https://mlsanity.com/paper/2601.02727/) (2026)
- [Difficulty-guided Sampling: Bridging the Target Gap between Dataset Distillation and Downstream Tasks](https://mlsanity.com/paper/2601.10090/) (2026)

## Abstract (arXiv)

Dataset distillation is an effective technique for reducing the cost and complexity of model training while maintaining performance by compressing large datasets into smaller, more efficient versions. In this paper, we present a novel generative dataset distillation method that can improve the accuracy of aligning prediction logits. Our approach integrates self-knowledge distillation to achieve more precise distribution matching between the synthetic and original data, thereby capturing the overall structure and relationships within the data. To further improve the accuracy of alignment, we introduce a standardization step on the logits before performing distribution matching, ensuring consistency in the range of logits. Through extensive experiments, we demonstrate that our method outperforms existing state-of-the-art methods, resulting in superior distillation performance.