# Elucidating the Design Space of Dataset Condensation

arXiv: https://arxiv.org/abs/2404.13733 · HTML: https://mlsanity.com/paper/2404.13733/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 56 references that are in the map.
- First posted on arXiv 21 Apr 2024.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022)
- [Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory](https://mlsanity.com/paper/2211.10586/) (2022)
- [Torchattacks: A PyTorch Repository for Adversarial Attacks](https://mlsanity.com/paper/2010.01950/) (2020)
- [Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective](https://mlsanity.com/paper/2306.13092/) (2023)
- [Data-free parameter pruning for Deep Neural Networks](https://mlsanity.com/paper/1507.06149/) (2015)
- [On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm](https://mlsanity.com/paper/2312.03526/) (2023)
- [Sharpness-Aware Minimization Improves Language Model Generalization](https://mlsanity.com/paper/2110.08529/) (2021)
- [Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data](https://mlsanity.com/paper/1912.07768/) (2019)

## Built on by (most cited first, 30 of 40)

- [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025)
- [Dataset Distillation via Committee Voting](https://mlsanity.com/paper/2501.07575/) (2025)
- [CaO2: Rectifying Inconsistencies in Diffusion-Based Dataset Distillation](https://mlsanity.com/paper/2506.22637/) (2025)
- [FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation](https://mlsanity.com/paper/2506.24125/) (2025)
- [DD-Ranking: Rethinking the Evaluation of Dataset Distillation](https://mlsanity.com/paper/2505.13300/) (2025)
- [MGD3: Mode-Guided Dataset Distillation using Diffusion Models](https://mlsanity.com/paper/2505.18963/) (2025)
- [Accelerating Diffusion Model Training under Minimal Budgets: A Condensation-Based Perspective](https://mlsanity.com/paper/2507.05914/) (2025)
- [Efficient Multimodal Dataset Distillation via Generative Models](https://mlsanity.com/paper/2509.15472/) (2025)
- [A Large-Scale Study on Video Action Dataset Condensation](https://mlsanity.com/paper/2412.21197/) (2024)
- [Unifying Dataset Pruning and Distillation for Efficient Large-scale Compression](https://mlsanity.com/paper/2502.06434/) (2025)
- [Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation](https://mlsanity.com/paper/2512.00308/) (2025)
- [Towards Consistent and Efficient Dataset Distillation via Diffusion-Driven Selection](https://mlsanity.com/paper/2412.09959/) (2024)
- [EVLF: Early Vision-Language Fusion for Generative Dataset Distillation](https://mlsanity.com/paper/2603.07476/) (2026)
- [Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling](https://mlsanity.com/paper/2511.18858/) (2025)
- [FD2: A Dedicated Framework for Fine-Grained Dataset Distillation](https://mlsanity.com/paper/2603.25144/) (2026)
- [Learnability-Guided Diffusion for Dataset Distillation](https://mlsanity.com/paper/2604.00519/) (2026)
- [Efficiency for Free: Ideal Data Are Transportable Representations](https://mlsanity.com/paper/2405.14669/) (2024)
- [Parameterizing Dataset Distillation via Gaussian Splatting](https://mlsanity.com/paper/2509.26219/) (2025)
- [Data Agent: Learning to Select Data via End-to-End Dynamic Optimization](https://mlsanity.com/paper/2603.07433/) (2026)
- [SCORE: Soft Label Compression-Centric Dataset Condensation via Coding Rate Optimization](https://mlsanity.com/paper/2503.13935/) (2025)
- [Temporal Saliency-Guided Distillation: A Scalable Framework for Distilling Video Datasets](https://mlsanity.com/paper/2505.20694/) (2025)
- [Rectified Decoupled Dataset Distillation: A Closer Look for Fair and Comprehensive Evaluation](https://mlsanity.com/paper/2509.19743/) (2025)
- [Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation](https://mlsanity.com/paper/2511.17914/) (2025)
- [DiRe: Diversity-promoting Regularization for Dataset Condensation](https://mlsanity.com/paper/2512.13083/) (2025)
- [Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift](https://mlsanity.com/paper/2512.15647/) (2025)
- [Rethinking Dataset Distillation: Hard Truths about Soft Labels](https://mlsanity.com/paper/2604.18811/) (2026)
- [Geometry-Aware Dataset Condensation for Diffusion Model Training](https://mlsanity.com/paper/2606.05883/) (2026)
- [Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation](https://mlsanity.com/paper/2606.20196/) (2026)
- [Condensing Large-Scale Datasets Directly with Minimal Information Loss](https://mlsanity.com/paper/2607.00916/) (2026)
- [Learning Robust Diffusion Models from Imprecise Supervision](https://mlsanity.com/paper/2510.03016/) (2025)

## Abstract (arXiv)

Dataset condensation, a concept within data-centric learning, efficiently transfers critical attributes from an original dataset to a synthetic version, maintaining both diversity and realism. This approach significantly improves model training efficiency and is adaptable across multiple application areas. Previous methods in dataset condensation have faced challenges: some incur high computational costs which limit scalability to larger datasets (e.g., MTT, DREAM, and TESLA), while others are restricted to less optimal design spaces, which could hinder potential improvements, especially in smaller datasets (e.g., SRe2L, G-VBSM, and RDED). To address these limitations, we propose a comprehensive design framework that includes specific, effective strategies like implementing soft category-aware matching and adjusting the learning rate schedule. These strategies are grounded in empirical evidence and theoretical backing. Our resulting approach, Elucidate Dataset Condensation (EDC), establishes a benchmark for both small and large-scale dataset condensation. In our testing, EDC achieves state-of-the-art accuracy, reaching 48.6% on ImageNet-1k with a ResNet-18 model at an IPC of 10, which corresponds to a compression ratio of 0.78%. This performance exceeds those of SRe2L, G-VBSM, and RDED by margins of 27.3%, 17.2%, and 6.6%, respectively.