Data to 14 Oct 2026

InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning

arXiv 2303.04947 ↗First posted 8 Mar 2023cs.CVcited by 90

Not on the map. Why?

Abstract (arXiv)

Data pruning aims to obtain lossless performances with less overall cost. A common approach is to filter out samples that make less contribution to the training. This could lead to gradient expectation bias compared to the original data. To solve this problem, we propose InfoBatch, a novel framework aiming to achieve lossless training acceleration by unbiased dynamic data pruning. Specifically, InfoBatch randomly prunes a portion of less informative samples based on the loss distribution and rescales the gradients of the remaining samples to approximate the original gradient. As a plug-and-play and architecture-agnostic framework, InfoBatch consistently obtains lossless training results on classification, semantic segmentation, vision pertaining, and instruction fine-tuning tasks. On CIFAR10/100, ImageNet-1K, and ADE20K, InfoBatch losslessly saves 40% overall cost. For pertaining MAE and diffusion model, InfoBatch can respectively save 24.8% and 27% cost. For LLaMA instruction fine-tuning, InfoBatch is also able to save 20% cost and is compatible with coreset selection methods. The code is publicly available at hrefhttps://github.com/henryqin1997/InfoBatchgithub.com/NUS-HPC-AI-Lab/InfoBatch.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Builds on 51 of its 85 references that are in the map. First posted on arXiv 8 Mar 2023.

Builds on

51 of 85 references.

References · 51

2015Deep Residual Learning for Image Recognitioncited by 52,096
2014Adam: A Method for Stochastic Optimizationcited by 37,240
2020Language Models are Few-Shot Learnerscited by 32,419
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045

38 more

2020Training data-efficient image transformers & distillation through attentioncited by 4,586
2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
2017Understanding Black-box Predictions via Influence Functionscited by 1,741
2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
2016An overview of gradient descent optimization algorithmscited by 1,033
2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
2017Large Batch Training of Convolutional Networkscited by 535
2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
2020Dataset Condensation with Gradient Matchingcited by 516
2022Dataset Distillation by Matching Training Trajectoriescited by 442
2021ResNet strikes back: An improved training procedure in timmcited by 368
2021Dataset Condensation with Distribution Matchingcited by 365
2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
2018An Empirical Model of Large-Batch Trainingcited by 281
2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 274
2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 226
2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 181
2018Adversarial Active Learning for Deep Networks: a Margin Based Approachcited by 158
2021Active Learning by Acquiring Contrastive Examplescited by 156
2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
2020Contextual Diversity for Active Learningcited by 139
2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 133
2018CurriculumNet: Weakly Supervised Learning from Large-Scale Web Imagescited by 126
2022Coverage-centric Coreset Selection for High Pruning Ratescited by 117
2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 94
2020Submodular Combinatorial Information Measures with Applications in Machine Learningcited by 88
2022Synthesizing Informative Training Samples with GANcited by 77
2021Accelerating Deep Learning with Dynamic Data Pruningcited by 43
2020Coresets for Near-Convex Functionscited by 38
2016Importance Sampling for Minibatchescited by 36

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 30; 2025: 21; 2026: 29.

90 papers.

Sort:

By area:

2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Cachingcited by 142
2024Rho-1: Not All Tokens Are What You Needcited by 133
2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
2023Dataset Quantizationcited by 48
2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 47
2024A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Trainingcited by 30
2024Navigating Complexity: Toward Lossless Graph Condensation via Expanding Window Matchingcited by 28
2023Too Large; Data Reduction for Vision-Language Pre-Trainingcited by 26
2025Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertaintycited by 22
2024A CLIP-Powered Framework for Robust and Generalizable Data Selectioncited by 21

38 more

2024EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Trainingcited by 18
2024Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Modelscited by 17
2023Dataset Condensation via Generative Modelcited by 15
2024Diversified Batch Selection for Training Accelerationcited by 14
2023Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraintscited by 13
2025LightLoc: Learning Outdoor LiDAR Localization at Light Speedcited by 10
2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
2024Beyond Efficiency: Molecular Data Pruning for Enhanced Generalizationcited by 9
2024Learnable Sparsity for Vision Generative Modelscited by 9
2025OASIS: Online Sample Selection for Continual Visual Instruction Tuningcited by 9
2024Two Trades is not Baffled: Condensing Graph via Crafting Rational Gradient Matchingcited by 8
2024GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruningcited by 8
2025Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuningcited by 8
2025Scale Efficient Training for Large Datasetscited by 7
2025Accelerating Diffusion Model Training under Minimal Budgets: A Condensation-Based Perspectivecited by 7
2023DREAM+: Efficient Dataset Distillation by Bidirectional Representative Matchingcited by 6
2023Data Optimization in Deep Learning: A Surveycited by 6
2024Language Model-Driven Data Pruning Enables Efficient Active Learningcited by 6
2024SCAN: Bootstrapping Contrastive Pre-training for Data Efficiencycited by 6
2025When Dynamic Data Selection Meets Data Augmentationcited by 6
2024Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimizationcited by 5
2024Data Lineage Inference: Uncovering Privacy Vulnerabilities of Dataset Pruningcited by 5
2024Dataset Growthcited by 4
2024CHG Shapley: Efficient Data Valuation and Selection towards Trustworthy Machine Learningcited by 4
2024Dynamic Data Pruning for Automatic Speech Recognitioncited by 4
2024Structural-Entropy-Based Sample Selection for Efficient and Effective Learningcited by 4
2024FLOPS: Forward Learning with OPtimal Samplingcited by 4
2025Progressive Data Dropout: An Embarrassingly Simple Approach to Faster Trainingcited by 4
2025Learning Sparsity for Effective and Efficient Music Performance Question Answeringcited by 4
2025Local Dense Logit Relations for Enhanced Knowledge Distillationcited by 4
2026ReMiT: RL-Guided Mid-Training for Iterative LLM Evolutioncited by 4
2025RL-Selector: Reinforcement Learning-Guided Data Selection via Redundancy Assessmentcited by 3
2025Partial Forward Blocking: A Novel Data Pruning Paradigm for Lossless Training Accelerationcited by 3
2025Differential-informed Sample Selection Accelerates Multimodal Contrastive Learningcited by 3
2024Exploring Learning Complexity for Efficient Downstream Dataset Pruningcited by 2
2025Layer-Aware Influence for Online Data Valuation Estimationcited by 2
2026Rethinking Representativeness and Diversity in Dynamic Data Selectioncited by 2
2026Data Agent: Learning to Select Data via End-to-End Dynamic Optimizationcited by 2

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way