Data to 7 Oct 2026

Accelerating Deep Learning by Focusing on the Biggest Losers

arXiv 1910.00762 ↗First posted 2 Oct 2019cs.LG stat.MLcited by 94

Not on the map. Why?

Abstract (arXiv)

This paper introduces Selective-Backprop, a technique that accelerates the training of deep neural networks (DNNs) by prioritizing examples with high loss at each iteration. Selective-Backprop uses the output of a training example's forward pass to decide whether to use that example to compute gradients and update parameters, or to skip immediately to the next example. By reducing the number of computationally-expensive backpropagation steps performed, Selective-Backprop accelerates training. Evaluation on CIFAR10, CIFAR100, and SVHN, across a variety of modern image models, shows that Selective-Backprop converges to target error rates up to 3.5x faster than with standard SGD and between 1.02--1.8x faster than a state-of-the-art importance sampling approach. Further acceleration of 26% can be achieved by using stale forward pass results for selection, thus also skipping forward passes of low priority examples.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 9; 2023: 23; 2024: 19; 2025: 16; 2026: 15.

94 papers.

Sort:

By area:

2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
2024Rho-1: Not All Tokens Are What You Needcited by 132
2024How to Train Data-Efficient LLMscited by 91
2021Chaos as an interpretable benchmark for forecasting and data-driven modellingcited by 87
2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 52
2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 44
2021On Tilted Losses in Machine Learning: Theory and Applicationscited by 42
2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37

38 more

2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
2021ACPL: Anti-curriculum Pseudo-labelling for Semi-supervised Medical Image Classificationcited by 34
2021Improving Contrastive Learning on Imbalanced Seed Data via Open-World Samplingcited by 30
2023Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understandingcited by 28
2022Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamicscited by 26
2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
2023Towards a statistical theory of data selection under weak supervisioncited by 25
2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
2024Swift Sampler: Efficient Learning of Sampler by 10 Parameterscited by 25
2022On Continual Model Refinement in Out-of-Distribution Data Streamscited by 22
2021Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrastcited by 21
2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
2025Learning to Reason at the Frontier of Learnabilitycited by 20
2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
2023Irreducible Curriculum for Language Model Pretrainingcited by 16
2025Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretrainingcited by 16
2025A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clonecited by 15
2023To Adapt or Not to Adapt? Real-Time Adaptation for Semantic Segmentationcited by 14
2024Diversified Batch Selection for Training Accelerationcited by 14
2025Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuningcited by 14
2021Improving Self-supervised Learning with Hardness-aware Dynamic Curriculum Learning: An Application to Digital Pathologycited by 13
2024Dataset Difficulty and the Role of Inductive Biascited by 12
2024Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalizationcited by 12
2024Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixturescited by 12
2023Towards Accelerated Model Training via Bayesian Data Selectioncited by 11
2022Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonizationcited by 9
2024Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitionscited by 7
2023PA&DA: Jointly Sampling PAth and DAta for Consistent NAScited by 6
2023Data Optimization in Deep Learning: A Surveycited by 6
2025Towards Active Synthetic Data Generation for Finetuning Language Modelscited by 6
2021How Important is Importance Sampling for Deep Budgeted Training?cited by 5
2022A Survey of Dataset Refinement for Problems in Computer Vision Datasetscited by 5
2022Accelerating Self-Supervised Learning via Efficient Training Strategiescited by 5
2023Influence Scores at Scale for Efficient Language Data Samplingcited by 5
2024Sketchy Moment Matching: Toward Fast and Provable Data Selection for Finetuningcited by 5
2021One Backward from Ten Forward, Subsampling for Large-Scale Deep Learningcited by 4
2021When does loss-based prioritization fail?cited by 4
2025DONOD: Efficient and Generalizable Instruction Fine-Tuning for LLMs via Model-Intrinsic Dataset Pruningcited by 4

50 most cited of 94 All 94 (JSON)