# Going Beyond Feature Similarity: Effective Dataset Distillation based on Class-Aware Conditional Mutual Information

arXiv: https://arxiv.org/abs/2412.09945 · HTML: https://mlsanity.com/paper/2412.09945/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 48 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022)
- [Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory](https://mlsanity.com/paper/2211.10586/) (2022)
- [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022)
- [Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective](https://mlsanity.com/paper/2306.13092/) (2023)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Improved Distribution Matching for Dataset Condensation](https://mlsanity.com/paper/2307.09742/) (2023)
- [Privacy for Free: How does Dataset Condensation Help Privacy?](https://mlsanity.com/paper/2206.00240/) (2022)
- [Coverage-centric Coreset Selection for High Pruning Rates](https://mlsanity.com/paper/2210.15809/) (2022)
- [Dataset Condensation with Contrastive Signals](https://mlsanity.com/paper/2202.02916/) (2022)
- [DREAM: Efficient Dataset Distillation by Representative Matching](https://mlsanity.com/paper/2302.14416/) (2023)
- [Efficient Dataset Distillation via Minimax Diffusion](https://mlsanity.com/paper/2311.15529/) (2023)
- [D4M: Dataset Distillation via Disentangled Diffusion Model](https://mlsanity.com/paper/2407.15138/) (2024)
- [MultiBooth: Towards Generating All Your Concepts in an Image from Text](https://mlsanity.com/paper/2404.14239/) (2024)
- [Sequential Subset Matching for Dataset Distillation](https://mlsanity.com/paper/2311.01570/) (2023)
- [InstantSwap: Fast Customized Concept Swapping across Sharp Shape Differences](https://mlsanity.com/paper/2412.01197/) (2024)
- [An Aggregation-Free Federated Learning for Tackling Data Heterogeneity](https://mlsanity.com/paper/2404.18962/) (2024)
- [Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment](https://mlsanity.com/paper/2409.17612/) (2024)
- [Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation](https://mlsanity.com/paper/2404.00563/) (2024)
- [Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information](https://mlsanity.com/paper/2401.08732/) (2024)

## Built on by (most cited first, 4 of 4)

- [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025)
- [Temporal Saliency-Guided Distillation: A Scalable Framework for Distilling Video Datasets](https://mlsanity.com/paper/2505.20694/) (2025)
- [Rectified Decoupled Dataset Distillation: A Closer Look for Fair and Comprehensive Evaluation](https://mlsanity.com/paper/2509.19743/) (2025)
- [Soft Label Pruning and Quantization for Large-Scale Dataset Distillation](https://mlsanity.com/paper/2604.18135/) (2026)

## Abstract (arXiv)

Dataset distillation (DD) aims to minimize the time and memory consumption needed for training deep neural networks on large datasets, by creating a smaller synthetic dataset that has similar performance to that of the full real dataset. However, current dataset distillation methods often result in synthetic datasets that are excessively difficult for networks to learn from, due to the compression of a substantial amount of information from the original data through metrics measuring feature similarity, e,g., distribution matching (DM). In this work, we introduce conditional mutual information (CMI) to assess the class-aware complexity of a dataset and propose a novel method by minimizing CMI. Specifically, we minimize the distillation loss while constraining the class-aware complexity of the synthetic dataset by minimizing its empirical CMI from the feature space of pre-trained networks, simultaneously. Conducting on a thorough set of experiments, we show that our method can serve as a general regularization method to existing DD methods and improve the performance and training efficiency.