# UDD: Dataset Distillation via Mining Underutilized Regions

arXiv: https://arxiv.org/abs/2408.16268 · HTML: https://mlsanity.com/paper/2408.16268/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 42 of its 59 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Coresets via Bilevel Optimization for Continual Learning and Streaming](https://mlsanity.com/paper/2006.03875/) (2020)
- [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022)
- [Salient Object Detection: A Benchmark](https://mlsanity.com/paper/1501.02741/) (2015)
- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [Online Coreset Selection for Rehearsal-based Continual Learning](https://mlsanity.com/paper/2106.01085/) (2021)
- [A Comprehensive Survey of Dataset Distillation](https://mlsanity.com/paper/2301.05603/) (2023)
- [Distilled One-Shot Federated Learning](https://mlsanity.com/paper/2009.07999/) (2020)
- [Soft-Label Dataset Distillation and Text Dataset Distillation](https://mlsanity.com/paper/1910.02551/) (2019)
- [Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data](https://mlsanity.com/paper/1912.07768/) (2019)
- [Salient Object Detection: A Survey](https://mlsanity.com/paper/1411.5878/) (2014)
- [DREAM: Efficient Dataset Distillation by Representative Matching](https://mlsanity.com/paper/2302.14416/) (2023)
- [Flexible Dataset Distillation: Learn Labels Instead of Images](https://mlsanity.com/paper/2006.08572/) (2020)

## Abstract (arXiv)

Dataset distillation synthesizes a small dataset such that a model trained on this set approximates the performance of the original dataset. Recent studies on dataset distillation focused primarily on the design of the optimization process, with methods such as gradient matching, feature alignment, and training trajectory matching. However, little attention has been given to the issue of underutilized regions in synthetic images. In this paper, we propose UDD, a novel approach to identify and exploit the underutilized regions to make them informative and discriminate, and thus improve the utilization of the synthetic dataset. Technically, UDD involves two underutilized regions searching policies for different conditions, i.e., response-based policy and data jittering-based policy. Compared with previous works, such two policies are utilization-sensitive, equipping with the ability to dynamically adjust the underutilized regions during the training process. Additionally, we analyze the current model optimization problem and design a category-wise feature contrastive loss, which can enhance the distinguishability of different categories and alleviate the shortcomings of the existing multi-formation methods. Experimentally, our method improves the utilization of the synthetic dataset and outperforms the state-of-the-art methods on various datasets, such as MNIST, FashionMNIST, SVHN, CIFAR-10, and CIFAR-100. For example, the improvements on CIFAR-10 and CIFAR-100 are 4.0% and 3.7% over the next best method with IPC=1, by mining the underutilized regions.