# Data Pruning via Moving-one-Sample-out

arXiv: https://arxiv.org/abs/2310.14664 · HTML: https://mlsanity.com/paper/2310.14664/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 51 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training](https://mlsanity.com/paper/2103.00123/) (2021)
- [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022)
- [Graph Condensation for Graph Neural Networks](https://mlsanity.com/paper/2110.07580/) (2021)
- [Dataset Pruning: Reducing Training Data by Examining Generalization Influence](https://mlsanity.com/paper/2205.09329/) (2022)
- [Online Coreset Selection for Rehearsal-based Continual Learning](https://mlsanity.com/paper/2106.01085/) (2021)
- [Coverage-centric Coreset Selection for High Pruning Rates](https://mlsanity.com/paper/2210.15809/) (2022)
- [Dataset Distillation via Factorization](https://mlsanity.com/paper/2210.16774/) (2022)

## Built on by (most cited first, 30 of 67)

- [On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm](https://mlsanity.com/paper/2312.03526/) (2023)
- [Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning](https://mlsanity.com/paper/2311.13613/) (2023)
- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [A CLIP-Powered Framework for Robust and Generalizable Data Selection](https://mlsanity.com/paper/2410.11215/) (2024)
- [Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learning](https://mlsanity.com/paper/2505.12212/) (2025)
- [Taming Diffusion for Dataset Distillation with High Representativeness](https://mlsanity.com/paper/2505.18399/) (2025)
- [How Far are AI-generated Videos from Simulating the 3D Visual World: A Learned 3D Evaluation Approach](https://mlsanity.com/paper/2406.19568/) (2024)
- [GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancers](https://mlsanity.com/paper/2503.19480/) (2025)
- [Data Pruning by Information Maximization](https://mlsanity.com/paper/2506.01701/) (2025)
- [Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillation](https://mlsanity.com/paper/2408.12483/) (2024)
- [Distilling the Knowledge in Data Pruning](https://mlsanity.com/paper/2403.07854/) (2024)
- [Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models](https://mlsanity.com/paper/2405.03869/) (2024)
- [Can OOD Object Detectors Learn from Foundation Models?](https://mlsanity.com/paper/2409.05162/) (2024)
- [Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness](https://mlsanity.com/paper/2412.06293/) (2024)
- [SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data Assimilation](https://mlsanity.com/paper/2409.13860/) (2024)
- [UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearning](https://mlsanity.com/paper/2502.15082/) (2025)
- [Training-Free Dataset Pruning for Instance Segmentation](https://mlsanity.com/paper/2503.00828/) (2025)
- [Towards More Diverse and Challenging Pre-training for Point Cloud Learning: Self-Supervised Cross Reconstruction with Decoupled Views](https://mlsanity.com/paper/2509.01250/) (2025)
- [Bridging Streaming Continual Learning via In-Context Large Tabular Models](https://mlsanity.com/paper/2512.11668/) (2025)
- [When Dynamic Data Selection Meets Data Augmentation](https://mlsanity.com/paper/2505.03809/) (2025)
- [Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring](https://mlsanity.com/paper/2506.08429/) (2025)
- [Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2410.10166/) (2024)
- [Data Lineage Inference: Uncovering Privacy Vulnerabilities of Dataset Pruning](https://mlsanity.com/paper/2411.15796/) (2024)
- [Equipping Vision Foundation Model with Mixture of Experts for Out-of-Distribution Detection](https://mlsanity.com/paper/2510.10584/) (2025)
- [Swift Cross-Dataset Pruning: Enhancing Fine-Tuning Efficiency in Natural Language Understanding](https://mlsanity.com/paper/2501.02432/) (2025)
- [RS-Prune: Training-Free Data Pruning at High Ratios for Efficient Remote Sensing Diffusion Foundation Models](https://mlsanity.com/paper/2512.23239/) (2025)
- [A Simple Remedy for Dataset Bias via Self-Influence: A Mislabeled Sample Perspective](https://mlsanity.com/paper/2411.00360/) (2024)
- [RL-Selector: Reinforcement Learning-Guided Data Selection via Redundancy Assessment](https://mlsanity.com/paper/2506.21037/) (2025)
- [Partial Forward Blocking: A Novel Data Pruning Paradigm for Lossless Training Acceleration](https://mlsanity.com/paper/2506.23674/) (2025)
- [AdaDeDup: Adaptive Hybrid Data Pruning for Efficient Large-Scale Object Detection Training](https://mlsanity.com/paper/2507.00049/) (2025)

## Abstract (arXiv)

In this paper, we propose a novel data-pruning approach called moving-one-sample-out (MoSo), which aims to identify and remove the least informative samples from the training set. The core insight behind MoSo is to determine the importance of each sample by assessing its impact on the optimal empirical risk. This is achieved by measuring the extent to which the empirical risk changes when a particular sample is excluded from the training set. Instead of using the computationally expensive leaving-one-out-retraining procedure, we propose an efficient first-order approximator that only requires gradient information from different training stages. The key idea behind our approximation is that samples with gradients that are consistently aligned with the average gradient of the training set are more informative and should receive higher scores, which could be intuitively understood as follows: if the gradient from a specific sample is consistent with the average gradient vector, it implies that optimizing the network using the sample will yield a similar effect on all remaining samples. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios and achieves satisfactory performance across various settings.