# Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation

arXiv: https://arxiv.org/abs/2404.00563 · HTML: https://mlsanity.com/paper/2404.00563/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 31 Mar 2024.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation](https://mlsanity.com/paper/2211.11004/) (2022)
- [Improved Distribution Matching for Dataset Condensation](https://mlsanity.com/paper/2307.09742/) (2023)
- [Privacy for Free: How does Dataset Condensation Help Privacy?](https://mlsanity.com/paper/2206.00240/) (2022)
- [Dataset Condensation with Contrastive Signals](https://mlsanity.com/paper/2202.02916/) (2022)
- [Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks](https://mlsanity.com/paper/2206.02916/) (2022)
- [Flexible Dataset Distillation: Learn Labels Instead of Images](https://mlsanity.com/paper/2006.08572/) (2020)
- [DataDAM: Efficient Dataset Distillation with Attention Matching](https://mlsanity.com/paper/2310.00093/) (2023)
- [Synthesizing Informative Training Samples with GAN](https://mlsanity.com/paper/2204.07513/) (2022)
- [DC-BENCH: Dataset Condensation Benchmark](https://mlsanity.com/paper/2207.09639/) (2022)
- [Distilled Replay: Overcoming Forgetting through Synthetic Samples](https://mlsanity.com/paper/2103.15851/) (2021)
- [Dataset Condensation with Latent Space Knowledge Factorization and Sharing](https://mlsanity.com/paper/2208.10494/) (2022)
- [Sample Condensation in Online Continual Learning](https://mlsanity.com/paper/2206.11849/) (2022)
- [No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"](https://mlsanity.com/paper/2209.14987/) (2022)

## Built on by (most cited first, 30 of 31)

- [Dataset Distillation with Neural Characteristic Function: A Minmax Perspective](https://mlsanity.com/paper/2502.20653/) (2025)
- [Elucidating the Design Space of Dataset Condensation](https://mlsanity.com/paper/2404.13733/) (2024)
- [Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment](https://mlsanity.com/paper/2409.17612/) (2024)
- [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025)
- [Taming Diffusion for Dataset Distillation with High Representativeness](https://mlsanity.com/paper/2505.18399/) (2025)
- [Hyperbolic Dataset Distillation](https://mlsanity.com/paper/2505.24623/) (2025)
- [Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator](https://mlsanity.com/paper/2408.06927/) (2024)
- [Generative Dataset Distillation Based on Diffusion Model](https://mlsanity.com/paper/2408.08610/) (2024)
- [Information-Guided Diffusion Sampling for Dataset Distillation](https://mlsanity.com/paper/2507.04619/) (2025)
- [HIERAMP: Coarse-to-Fine Autoregressive Amplification for Generative Dataset Distillation](https://mlsanity.com/paper/2603.06932/) (2026)
- [Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation](https://mlsanity.com/paper/2505.14705/) (2025)
- [Efficient Multimodal Dataset Distillation via Generative Models](https://mlsanity.com/paper/2509.15472/) (2025)
- [Decoupled Audio-Visual Dataset Distillation](https://mlsanity.com/paper/2511.17890/) (2025)
- [Dataset Condensation with Color Compensation](https://mlsanity.com/paper/2508.01139/) (2025)
- [Going Beyond Feature Similarity: Effective Dataset Distillation based on Class-Aware Conditional Mutual Information](https://mlsanity.com/paper/2412.09945/) (2024)
- [Dataset Distillation from First Principles: Integrating Core Information Extraction and Purposeful Learning](https://mlsanity.com/paper/2409.01410/) (2024)
- [Decomposed Distribution Matching in Dataset Condensation](https://mlsanity.com/paper/2412.04748/) (2024)
- [Diffusion Models as Dataset Distillation Priors](https://mlsanity.com/paper/2510.17421/) (2025)
- [GeoDM: Geometry-aware Distribution Matching for Dataset Distillation](https://mlsanity.com/paper/2512.08317/) (2025)
- [Multimodal Distribution Matching for Vision-Language Dataset Distillation](https://mlsanity.com/paper/2605.23482/) (2026)
- [Understanding Dataset Distillation via Spectral Filtering](https://mlsanity.com/paper/2503.01212/) (2025)
- [Robust Dataset Distillation by Matching Adversarial Trajectories](https://mlsanity.com/paper/2503.12069/) (2025)
- [Instance Data Condensation for Image Super-Resolution](https://mlsanity.com/paper/2505.21099/) (2025)
- [DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery](https://mlsanity.com/paper/2605.12649/) (2026)
- [Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation](https://mlsanity.com/paper/2606.20196/) (2026)
- [Dataset Distillation of 3D Point Clouds via Distribution Matching](https://mlsanity.com/paper/2503.22154/) (2025)
- [Distill Video Datasets into Images](https://mlsanity.com/paper/2512.14621/) (2025)
- [Foreground-Aware Dataset Distillation via Dynamic Patch Selection](https://mlsanity.com/paper/2601.02727/) (2026)
- [Towards Principled Dataset Distillation: A Spectral Distribution Perspective](https://mlsanity.com/paper/2603.01698/) (2026)
- [Memory-efficient Continual Learning with Prototypical Exemplar Condensation](https://mlsanity.com/paper/2603.13804/) (2026)

## Abstract (arXiv)

Dataset distillation has emerged as a promising approach in deep learning, enabling efficient training with small synthetic datasets derived from larger real ones. Particularly, distribution matching-based distillation methods attract attention thanks to its effectiveness and low computational cost. However, these methods face two primary limitations: the dispersed feature distribution within the same class in synthetic datasets, reducing class discrimination, and an exclusive focus on mean feature consistency, lacking precision and comprehensiveness. To address these challenges, we introduce two novel constraints: a class centralization constraint and a covariance matching constraint. The class centralization constraint aims to enhance class discrimination by more closely clustering samples within classes. The covariance matching constraint seeks to achieve more accurate feature distribution matching between real and synthetic datasets through local feature covariance matrices, particularly beneficial when sample sizes are much smaller than the number of features. Experiments demonstrate notable improvements with these constraints, yielding performance boosts of up to 6.6% on CIFAR10, 2.9% on SVHN, 2.5% on CIFAR100, and 2.5% on TinyImageNet, compared to the state-of-the-art relevant methods. In addition, our method maintains robust performance in cross-architecture settings, with a maximum performance drop of 1.7% on four architectures. Code is available at https://github.com/VincenDen/IID.