# Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning

arXiv: https://arxiv.org/abs/2311.13613 · HTML: https://mlsanity.com/paper/2311.13613/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 58 references that are in the map.
- First posted on arXiv 22 Nov 2023.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training](https://mlsanity.com/paper/2103.00123/) (2021)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02934/) (2021)
- [Active Learning by Acquiring Contrastive Examples](https://mlsanity.com/paper/2109.03764/) (2021)
- [DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning](https://mlsanity.com/paper/2204.08499/) (2022)
- [Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation](https://mlsanity.com/paper/2211.11004/) (2022)
- [Efficient Sharpness-aware Minimization for Improved Training of Neural Networks](https://mlsanity.com/paper/2110.03141/) (2021)
- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [Online Batch Selection for Faster Training of Neural Networks](https://mlsanity.com/paper/1511.06343/) (2015)
- [Coverage-centric Coreset Selection for High Pruning Rates](https://mlsanity.com/paper/2210.15809/) (2022)
- [Estimating Example Difficulty Using Variance of Gradients](https://mlsanity.com/paper/2008.11600/) (2020)
- [Data Pruning via Moving-one-Sample-out](https://mlsanity.com/paper/2310.14664/) (2023)
- [Large-scale Dataset Pruning with Dynamic Uncertainty](https://mlsanity.com/paper/2306.05175/) (2023)

## Built on by (most cited first, 30 of 38)

- [Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment](https://mlsanity.com/paper/2409.17612/) (2024)
- [Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty](https://mlsanity.com/paper/2502.06905/) (2025)
- [A CLIP-Powered Framework for Robust and Generalizable Data Selection](https://mlsanity.com/paper/2410.11215/) (2024)
- [Taming Diffusion for Dataset Distillation with High Representativeness](https://mlsanity.com/paper/2505.18399/) (2025)
- [Diversified Batch Selection for Training Acceleration](https://mlsanity.com/paper/2406.04872/) (2024)
- [Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator](https://mlsanity.com/paper/2408.06927/) (2024)
- [Zero-Shot Coreset Selection via Iterative Subspace Sampling](https://mlsanity.com/paper/2411.15349/) (2024)
- [Unifying Dataset Pruning and Distillation for Efficient Large-scale Compression](https://mlsanity.com/paper/2502.06434/) (2025)
- [When Dynamic Data Selection Meets Data Augmentation](https://mlsanity.com/paper/2505.03809/) (2025)
- [Position: AI Scaling: From Up to Down and Out](https://mlsanity.com/paper/2502.01677/) (2025)
- [Distributional Machine Unlearning via Selective Data Removal](https://mlsanity.com/paper/2507.15112/) (2025)
- [A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Design](https://mlsanity.com/paper/2607.09084/) (2026)
- [RS-Prune: Training-Free Data Pruning at High Ratios for Efficient Remote Sensing Diffusion Foundation Models](https://mlsanity.com/paper/2512.23239/) (2025)
- [RL-Selector: Reinforcement Learning-Guided Data Selection via Redundancy Assessment](https://mlsanity.com/paper/2506.21037/) (2025)
- [Coresets from Trajectories: Selecting Data via Correlation of Loss Differences](https://mlsanity.com/paper/2508.20230/) (2025)
- [Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling](https://mlsanity.com/paper/2510.20673/) (2025)
- [Learnability-Guided Diffusion for Dataset Distillation](https://mlsanity.com/paper/2604.00519/) (2026)
- [Effective Data Pruning through Score Extrapolation](https://mlsanity.com/paper/2506.09010/) (2025)
- [Class-Proportional Coreset Selection for Difficulty-Separable Data](https://mlsanity.com/paper/2507.10904/) (2025)
- [Data Agent: Learning to Select Data via End-to-End Dynamic Optimization](https://mlsanity.com/paper/2603.07433/) (2026)
- [Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning](https://mlsanity.com/paper/2507.12750/) (2025)
- [On-the-Fly Data Augmentation via Gradient-Guided and Sample-Aware Influence Estimation](https://mlsanity.com/paper/2510.00434/) (2025)
- [DiRe: Diversity-promoting Regularization for Dataset Condensation](https://mlsanity.com/paper/2512.13083/) (2025)
- [Geometry-Aware Dataset Condensation for Diffusion Model Training](https://mlsanity.com/paper/2606.05883/) (2026)
- [Are Coreset Selection Methods Worth Their Cost?](https://mlsanity.com/paper/2609.22894/) (2026)
- [DiffProb: Data Pruning for Face Recognition](https://mlsanity.com/paper/2505.15272/) (2025)
- [Extending Dataset Pruning to Object Detection: A Variance-based Approach](https://mlsanity.com/paper/2505.17245/) (2025)
- [Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection](https://mlsanity.com/paper/2506.14473/) (2025)
- [dreaMLearning: Data Compression Assisted Machine Learning](https://mlsanity.com/paper/2506.22190/) (2025)
- [UNSEEN: Enhancing Dataset Pruning from a Generalization Perspective](https://mlsanity.com/paper/2511.12988/) (2025)

## Abstract (arXiv)

Dataset pruning aims to construct a coreset capable of achieving performance comparable to the original, full dataset. Most existing dataset pruning methods rely on snapshot-based criteria to identify representative samples, often resulting in poor generalization across various pruning and cross-architecture scenarios. Recent studies have addressed this issue by expanding the scope of training dynamics considered, including factors such as forgetting event and probability change, typically using an averaging approach. However, these works struggle to integrate a broader range of training dynamics without overlooking well-generalized samples, which may not be sufficiently highlighted in an averaging manner. In this study, we propose a novel dataset pruning method termed as Temporal Dual-Depth Scoring (TDDS), to tackle this problem. TDDS utilizes a dual-depth strategy to achieve a balance between incorporating extensive training dynamics and identifying representative samples for dataset pruning. In the first depth, we estimate the series of each sample's individual contributions spanning the training progress, ensuring comprehensive integration of training dynamics. In the second depth, we focus on the variability of the sample-wise contributions identified in the first depth to highlight well-generalized samples. Extensive experiments conducted on CIFAR and ImageNet datasets verify the superiority of TDDS over previous SOTA methods. Specifically on CIFAR-100, our method achieves 54.51% accuracy with only 10% training data, surpassing random selection by 7.83% and other comparison methods by at least 12.69%.