# Importance-Aware Adaptive Dataset Distillation

arXiv: https://arxiv.org/abs/2401.15863 · HTML: https://mlsanity.com/paper/2401.15863/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 91 references that are in the map.
- First posted on arXiv 29 Jan 2024.

## Builds on (most cited first, 30 of 65)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training](https://mlsanity.com/paper/2103.00123/) (2021)
- [Recent Trends in Deep Learning Based Natural Language Processing](https://mlsanity.com/paper/1708.02709/) (2017)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning](https://mlsanity.com/paper/2012.10630/) (2020)
- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022)
- [Integrating Scientific Knowledge with Machine Learning for Engineering and Environmental Systems](https://mlsanity.com/paper/2003.04919/) (2020)
- [Graph Condensation for Graph Neural Networks](https://mlsanity.com/paper/2110.07580/) (2021)

## Built on by (most cited first, 18 of 18)

- [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025)
- [Generative Dataset Distillation: Balancing Global Structure and Local Details](https://mlsanity.com/paper/2404.17732/) (2024)
- [Hyperbolic Dataset Distillation](https://mlsanity.com/paper/2505.24623/) (2025)
- [Generative Dataset Distillation Based on Diffusion Model](https://mlsanity.com/paper/2408.08610/) (2024)
- [Diversity-Driven Generative Dataset Distillation Based on Diffusion Model with Self-Adaptive Memory](https://mlsanity.com/paper/2505.19469/) (2025)
- [Information-Guided Diffusion Sampling for Dataset Distillation](https://mlsanity.com/paper/2507.04619/) (2025)
- [Applications of Knowledge Distillation in Remote Sensing: A Survey](https://mlsanity.com/paper/2409.12111/) (2024)
- [Task-Specific Generative Dataset Distillation with Difficulty-Guided Sampling](https://mlsanity.com/paper/2507.03331/) (2025)
- [Decoupled Audio-Visual Dataset Distillation](https://mlsanity.com/paper/2511.17890/) (2025)
- [Generative Dataset Distillation Based on Self-knowledge Distillation](https://mlsanity.com/paper/2501.04202/) (2025)
- [FD2: A Dedicated Framework for Fine-Grained Dataset Distillation](https://mlsanity.com/paper/2603.25144/) (2026)
- [Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models](https://mlsanity.com/paper/2605.07194/) (2026)
- [UDD: Dataset Distillation via Mining Underutilized Regions](https://mlsanity.com/paper/2408.16268/) (2024)
- [Foreground-Aware Dataset Distillation via Dynamic Patch Selection](https://mlsanity.com/paper/2601.02727/) (2026)
- [Difficulty-guided Sampling: Bridging the Target Gap between Dataset Distillation and Downstream Tasks](https://mlsanity.com/paper/2601.10090/) (2026)
- [SAS: Semantic-aware Sampling for Generative Dataset Distillation](https://mlsanity.com/paper/2605.18012/) (2026)
- [DeCO: Discriminative Evidence Composition for Fine-Grained Dataset Distillation](https://mlsanity.com/paper/2608.25480/) (2026)
- [PSM: Dataset Distillation Based on Precise Statistical Matching by Difficulty](https://mlsanity.com/paper/2609.34299/) (2026)

## Abstract (arXiv)

Herein, we propose a novel dataset distillation method for constructing small informative datasets that preserve the information of the large original datasets. The development of deep learning models is enabled by the availability of large-scale datasets. Despite unprecedented success, large-scale datasets considerably increase the storage and transmission costs, resulting in a cumbersome model training process. Moreover, using raw data for training raises privacy and copyright concerns. To address these issues, a new task named dataset distillation has been introduced, aiming to synthesize a compact dataset that retains the essential information from the large original dataset. State-of-the-art (SOTA) dataset distillation methods have been proposed by matching gradients or network parameters obtained during training on real and synthetic datasets. The contribution of different network parameters to the distillation process varies, and uniformly treating them leads to degraded distillation performance. Based on this observation, we propose an importance-aware adaptive dataset distillation (IADD) method that can improve distillation performance by automatically assigning importance weights to different network parameters during distillation, thereby synthesizing more robust distilled datasets. IADD demonstrates superior performance over other SOTA dataset distillation methods based on parameter matching on multiple benchmark datasets and outperforms them in terms of cross-architecture generalization. In addition, the analysis of self-adaptive weights demonstrates the effectiveness of IADD. Furthermore, the effectiveness of IADD is validated in a real-world medical application such as COVID-19 detection.