# Dataset Distillation via the Wasserstein Metric

arXiv: https://arxiv.org/abs/2311.18531 · HTML: https://mlsanity.com/paper/2311.18531/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 65 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [Fast Computation of Wasserstein Barycenters](https://mlsanity.com/paper/1310.4375/) (2013)
- [Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory](https://mlsanity.com/paper/2211.10586/) (2022)
- [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022)
- [Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation](https://mlsanity.com/paper/2211.11004/) (2022)
- [Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective](https://mlsanity.com/paper/2306.13092/) (2023)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Privacy for Free: How does Dataset Condensation Help Privacy?](https://mlsanity.com/paper/2206.00240/) (2022)
- [Dataset Distillation via Factorization](https://mlsanity.com/paper/2210.16774/) (2022)
- [A Comprehensive Survey of Dataset Distillation](https://mlsanity.com/paper/2301.05603/) (2023)
- [Efficient Dataset Distillation Using Random Feature Approximation](https://mlsanity.com/paper/2210.12067/) (2022)
- [Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data](https://mlsanity.com/paper/1912.07768/) (2019)
- [Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks](https://mlsanity.com/paper/2206.02916/) (2022)
- [DENSE: Data-Free One-Shot Federated Learning](https://mlsanity.com/paper/2112.12371/) (2021)
- [DataDAM: Efficient Dataset Distillation with Attention Matching](https://mlsanity.com/paper/2310.00093/) (2023)
- [Synthesizing Informative Training Samples with GAN](https://mlsanity.com/paper/2204.07513/) (2022)
- [Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching](https://mlsanity.com/paper/2311.17950/) (2023)
- [Federated Learning via Synthetic Data](https://mlsanity.com/paper/2008.04489/) (2020)
- [Scalable Computations of Wasserstein Barycenter via Input Convex Neural Networks](https://mlsanity.com/paper/2007.04462/) (2020)
- [A Fast Knowledge Distillation Framework for Visual Recognition](https://mlsanity.com/paper/2112.01528/) (2021)

## Built on by (most cited first, 30 of 31)

- [Elucidating the Design Space of Dataset Condensation](https://mlsanity.com/paper/2404.13733/) (2024)
- [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025)
- [DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation](https://mlsanity.com/paper/2411.19946/) (2024)
- [Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios](https://mlsanity.com/paper/2410.17193/) (2024)
- [Towards Adversarially Robust Dataset Distillation by Curvature Regularization](https://mlsanity.com/paper/2403.10045/) (2024)
- [Self-supervised Dataset Distillation: A Good Compression Is All You Need](https://mlsanity.com/paper/2404.07976/) (2024)
- [Distributional Dataset Distillation with Subtask Decomposition](https://mlsanity.com/paper/2403.00999/) (2024)
- [Efficient Multimodal Dataset Distillation via Generative Models](https://mlsanity.com/paper/2509.15472/) (2025)
- [CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecasting](https://mlsanity.com/paper/2406.02131/) (2024)
- [Dataset Condensation with Latent Quantile Matching](https://mlsanity.com/paper/2406.09860/) (2024)
- [Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation](https://mlsanity.com/paper/2512.00308/) (2025)
- [Prioritize Alignment in Dataset Distillation](https://mlsanity.com/paper/2408.03360/) (2024)
- [GeoDM: Geometry-aware Distribution Matching for Dataset Distillation](https://mlsanity.com/paper/2512.08317/) (2025)
- [DMGD: Train-Free Dataset Distillation with Semantic-Distribution Matching in Diffusion Models](https://mlsanity.com/paper/2605.03877/) (2026)
- [Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models](https://mlsanity.com/paper/2605.07194/) (2026)
- [Multimodal Distribution Matching for Vision-Language Dataset Distillation](https://mlsanity.com/paper/2605.23482/) (2026)
- [UniPROT: Uniform Prototype Selection via Partial Optimal Transport with Submodular Guarantees](https://mlsanity.com/paper/2604.10952/) (2026)
- [Dataset Distillation as Pushforward Optimal Quantization](https://mlsanity.com/paper/2501.07681/) (2025)
- [A Discrepancy-Based Perspective on Dataset Condensation](https://mlsanity.com/paper/2509.10367/) (2025)
- [Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation](https://mlsanity.com/paper/2606.20196/) (2026)
- [Structural Assessment for Understanding and Guiding Dataset Distillation in Discrete Token Space](https://mlsanity.com/paper/2606.21705/) (2026)
- [Condensing Large-Scale Datasets Directly with Minimal Information Loss](https://mlsanity.com/paper/2607.00916/) (2026)
- [KD2M: A unifying framework for feature knowledge distillation](https://mlsanity.com/paper/2504.01757/) (2025)
- [Distill Video Datasets into Images](https://mlsanity.com/paper/2512.14621/) (2025)
- [Difficulty-guided Sampling: Bridging the Target Gap between Dataset Distillation and Downstream Tasks](https://mlsanity.com/paper/2601.10090/) (2026)
- [Towards Principled Dataset Distillation: A Spectral Distribution Perspective](https://mlsanity.com/paper/2603.01698/) (2026)
- [HFP-SAM: Hierarchical Frequency Prompted SAM for Efficient Marine Animal Segmentation](https://mlsanity.com/paper/2603.12708/) (2026)
- [Omnimodal Dataset Distillation via High-order Proxy Alignment](https://mlsanity.com/paper/2604.10666/) (2026)
- [Soft Label Pruning and Quantization for Large-Scale Dataset Distillation](https://mlsanity.com/paper/2604.18135/) (2026)
- [Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation](https://mlsanity.com/paper/2606.29464/) (2026)

## Abstract (arXiv)

Dataset Distillation (DD) aims to generate a compact synthetic dataset that enables models to achieve performance comparable to training on the full large dataset, significantly reducing computational costs. Drawing from optimal transport theory, we introduce WMDD (Wasserstein Metric-based Dataset Distillation), a straightforward yet powerful method that employs the Wasserstein metric to enhance distribution matching. We compute the Wasserstein barycenter of features from a pretrained classifier to capture essential characteristics of the original data distribution. By optimizing synthetic data to align with this barycenter in feature space and leveraging per-class BatchNorm statistics to preserve intra-class variations, WMDD maintains the efficiency of distribution matching approaches while achieving state-of-the-art results across various high-resolution datasets. Our extensive experiments demonstrate WMDD's effectiveness and adaptability, highlighting its potential for advancing machine learning applications at scale.