# Dataset Distillation by Automatic Training Trajectories arXiv: https://arxiv.org/abs/2407.14245 ยท HTML: https://mlsanity.com/paper/2407.14245/ ## Key facts - Cited by 26 papers in the map (data through 14 Oct 2026). - Builds on 48 of its 71 references that are in the map. - First posted on arXiv 19 Jul 2024. ## Builds on (most cited first, 30 of 48) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017) - [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016) - [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018) - [Green AI](https://mlsanity.com/paper/1907.10597/) (2019) - [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019) - [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021) - [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022) - [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018) - [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021) - [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021) - [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018) - [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021) - [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022) - [Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory](https://mlsanity.com/paper/2211.10586/) (2022) - [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022) - [Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation](https://mlsanity.com/paper/2211.11004/) (2022) - [Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective](https://mlsanity.com/paper/2306.13092/) (2023) - [Dataset Pruning: Reducing Training Data by Examining Generalization Influence](https://mlsanity.com/paper/2205.09329/) (2022) - [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023) - [Improved Distribution Matching for Dataset Condensation](https://mlsanity.com/paper/2307.09742/) (2023) - [Privacy for Free: How does Dataset Condensation Help Privacy?](https://mlsanity.com/paper/2206.00240/) (2022) - [Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching](https://mlsanity.com/paper/2310.05773/) (2023) - [Online Coreset Selection for Rehearsal-based Continual Learning](https://mlsanity.com/paper/2106.01085/) (2021) - [Distilled One-Shot Federated Learning](https://mlsanity.com/paper/2009.07999/) (2020) - [Soft-Label Dataset Distillation and Text Dataset Distillation](https://mlsanity.com/paper/1910.02551/) (2019) - [Dataset Condensation with Contrastive Signals](https://mlsanity.com/paper/2202.02916/) (2022) - [Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks](https://mlsanity.com/paper/2206.02916/) (2022) ## Built on by (most cited first, 26 of 26) - [Dataset Distillation with Neural Characteristic Function: A Minmax Perspective](https://mlsanity.com/paper/2502.20653/) (2025) - [Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment](https://mlsanity.com/paper/2409.17612/) (2024) - [The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions](https://mlsanity.com/paper/2502.05673/) (2025) - [Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarios](https://mlsanity.com/paper/2410.17193/) (2024) - [Generative Dataset Distillation Based on Diffusion Model](https://mlsanity.com/paper/2408.08610/) (2024) - [DD-Ranking: Rethinking the Evaluation of Dataset Distillation](https://mlsanity.com/paper/2505.13300/) (2025) - [Information-Guided Diffusion Sampling for Dataset Distillation](https://mlsanity.com/paper/2507.04619/) (2025) - [Beyond Random: Automatic Inner-loop Optimization in Dataset Distillation](https://mlsanity.com/paper/2510.04838/) (2025) - [Prioritize Alignment in Dataset Distillation](https://mlsanity.com/paper/2408.03360/) (2024) - [Dataset Distillation from First Principles: Integrating Core Information Extraction and Purposeful Learning](https://mlsanity.com/paper/2409.01410/) (2024) - [GeoDM: Geometry-aware Distribution Matching for Dataset Distillation](https://mlsanity.com/paper/2512.08317/) (2025) - [DMGD: Train-Free Dataset Distillation with Semantic-Distribution Matching in Diffusion Models](https://mlsanity.com/paper/2605.03877/) (2026) - [Multimodal Distribution Matching for Vision-Language Dataset Distillation](https://mlsanity.com/paper/2605.23482/) (2026) - [High-Order Progressive Trajectory Matching for Medical Image Dataset Distillation](https://mlsanity.com/paper/2509.24177/) (2025) - [Parameterizing Dataset Distillation via Gaussian Splatting](https://mlsanity.com/paper/2509.26219/) (2025) - [Learning from Dense Events: Towards Fast Spiking Neural Networks Training via Event Dataset Distillation](https://mlsanity.com/paper/2511.12095/) (2025) - [Robust Dataset Distillation by Matching Adversarial Trajectories](https://mlsanity.com/paper/2503.12069/) (2025) - [Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation](https://mlsanity.com/paper/2511.17914/) (2025) - [C2TC: A Training-Free Framework for Efficient Tabular Data Condensation](https://mlsanity.com/paper/2602.21717/) (2026) - [Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation](https://mlsanity.com/paper/2606.20196/) (2026) - [Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining](https://mlsanity.com/paper/2608.13515/) (2026) - [Dataset Distillation of 3D Point Clouds via Distribution Matching](https://mlsanity.com/paper/2503.22154/) (2025) - [Label-Consistent Dataset Distillation with Detector-Guided Refinement](https://mlsanity.com/paper/2507.13074/) (2025) - [Omnimodal Dataset Distillation via High-order Proxy Alignment](https://mlsanity.com/paper/2604.10666/) (2026) - [Fair Dataset Distillation via Cross-Group Barycenter Alignment](https://mlsanity.com/paper/2605.00185/) (2026) - [Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation](https://mlsanity.com/paper/2606.29464/) (2026) ## Abstract (arXiv) Dataset Distillation is used to create a concise, yet informative, synthetic dataset that can replace the original dataset for training purposes. Some leading methods in this domain prioritize long-range matching, involving the unrolling of training trajectories with a fixed number of steps (NS) on the synthetic dataset to align with various expert training trajectories. However, traditional long-range matching methods possess an overfitting-like problem, the fixed step size NS forces synthetic dataset to distortedly conform seen expert training trajectories, resulting in a loss of generality-especially to those from unencountered architecture. We refer to this as the Accumulated Mismatching Problem (AMP), and propose a new approach, Automatic Training Trajectories (ATT), which dynamically and adaptively adjusts trajectory length NS to address the AMP. Our method outperforms existing methods particularly in tests involving cross-architectures. Moreover, owing to its adaptive nature, it exhibits enhanced stability in the face of parameter variations.