# Group Distributionally Robust Dataset Distillation with Risk Minimization arXiv: https://arxiv.org/abs/2402.04676 ยท HTML: https://mlsanity.com/paper/2402.04676/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 45 references that are in the map. - First posted on arXiv 7 Feb 2024. ## Builds on (most cited first, 30 of 30) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022) - [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018) - [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021) - [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018) - [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021) - [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021) - [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022) - [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022) - [Distributionally Robust Language Modeling](https://mlsanity.com/paper/1909.02060/) (2019) - [Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation](https://mlsanity.com/paper/2211.11004/) (2022) - [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023) - [Improved Distribution Matching for Dataset Condensation](https://mlsanity.com/paper/2307.09742/) (2023) - [Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data](https://mlsanity.com/paper/1912.07768/) (2019) - [DREAM: Efficient Dataset Distillation by Representative Matching](https://mlsanity.com/paper/2302.14416/) (2023) - [DataDAM: Efficient Dataset Distillation with Attention Matching](https://mlsanity.com/paper/2310.00093/) (2023) - [Efficient Dataset Distillation via Minimax Diffusion](https://mlsanity.com/paper/2311.15529/) (2023) - [A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization](https://mlsanity.com/paper/1905.01332/) (2019) - [Dataset Distillation with Convexified Implicit Gradients](https://mlsanity.com/paper/2302.06755/) (2023) - [DiM: Distilling Dataset into Generative Model](https://mlsanity.com/paper/2303.04707/) (2023) - [From Data to Decisions: Distributionally Robust Optimization is Optimal](https://mlsanity.com/paper/1704.04118/) (2017) - [Meta Knowledge Condensation for Federated Learning](https://mlsanity.com/paper/2209.14851/) (2022) - [Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents](https://mlsanity.com/paper/2312.01537/) (2023) - [PAC-Bayesian Bound for the Conditional Value at Risk](https://mlsanity.com/paper/2006.14763/) (2020) - [Generalization Bounds with Minimal Dependency on Hypothesis Class via Distributionally Robust Optimization](https://mlsanity.com/paper/2106.11180/) (2021) - [Towards Robust Dataset Learning](https://mlsanity.com/paper/2211.10752/) (2022) - [Federated Distributionally Robust Optimization with Non-Convex Objectives: Algorithm and Analysis](https://mlsanity.com/paper/2307.14364/) (2023) - [Group Distributionally Robust Knowledge Distillation](https://mlsanity.com/paper/2311.00476/) (2023) ## Built on by (most cited first, 10 of 10) - [Towards Adversarially Robust Dataset Distillation by Curvature Regularization](https://mlsanity.com/paper/2403.10045/) (2024) - [HIERAMP: Coarse-to-Fine Autoregressive Amplification for Generative Dataset Distillation](https://mlsanity.com/paper/2603.06932/) (2026) - [Dataset Distillation from First Principles: Integrating Core Information Extraction and Purposeful Learning](https://mlsanity.com/paper/2409.01410/) (2024) - [FairDD: Fair Dataset Distillation](https://mlsanity.com/paper/2411.19623/) (2024) - [CONCORD: Concept-Informed Diffusion for Dataset Distillation](https://mlsanity.com/paper/2505.18358/) (2025) - [Dataset Distillation as Pushforward Optimal Quantization](https://mlsanity.com/paper/2501.07681/) (2025) - [HSFM: Hard-Set-Guided Feature-Space Meta-Learning for Robust Classification under Spurious Correlations](https://mlsanity.com/paper/2603.29313/) (2026) - [Structural Assessment for Understanding and Guiding Dataset Distillation in Discrete Token Space](https://mlsanity.com/paper/2606.21705/) (2026) - [Beyond Discrete Samples: High Information Density Replay for Efficient Lifelong Person Re-Identification](https://mlsanity.com/paper/2508.01587/) (2025) - [Fair Dataset Distillation via Cross-Group Barycenter Alignment](https://mlsanity.com/paper/2605.00185/) (2026) ## Abstract (arXiv) Dataset distillation (DD) has emerged as a widely adopted technique for crafting a synthetic dataset that captures the essential information of a training dataset, facilitating the training of accurate neural models. Its applications span various domains, including transfer learning, federated learning, and neural architecture search. The most popular methods for constructing the synthetic data rely on matching the convergence properties of training the model with the synthetic dataset and the training dataset. However, using the empirical loss as the criterion must be thought of as auxiliary in the same sense that the training set is an approximate substitute for the population distribution, and the latter is the data of interest. Yet despite its popularity, an aspect that remains unexplored is the relationship of DD to its generalization, particularly across uncommon subgroups. That is, how can we ensure that a model trained on the synthetic dataset performs well when faced with samples from regions with low population density? Here, the representativeness and coverage of the dataset become salient over the guaranteed training error at inference. Drawing inspiration from distributionally robust optimization, we introduce an algorithm that combines clustering with the minimization of a risk measure on the loss to conduct DD. We provide a theoretical rationale for our approach and demonstrate its effective generalization and robustness across subgroups through numerical experiments. The source code is available at https://github.com/Mming11/RobustDatasetDistillation.