Data to 7 Oct 2026

Not All Samples Are Created Equal: Deep Learning with Importance Sampling

arXiv 1803.00942 ↗First posted 2 Mar 2018cs.LGcited by 323

Not on the map. Why?

Abstract (arXiv)

Deep neural network training spends most of the computation on examples that are properly handled, and could be ignored. We propose to mitigate this phenomenon with a principled importance sampling scheme that focuses computation on "informative" examples, and reduces the variance of the stochastic gradients during training. Our contribution is twofold: first, we derive a tractable upper bound to the per-sample gradient norm, and second we derive an estimator of the variance reduction achieved with importance sampling, which enables us to switch it on when it will result in an actual speedup. The resulting scheme can be used by changing a few lines of code in a standard SGD procedure, and we demonstrate experimentally, on image classification, CNN fine-tuning, and RNN training, that for a fixed wall-clock time budget, it provides a reduction of the train losses of up to an order of magnitude and a relative improvement of test errors between 5% and 17%.

Key facts. Cited by 323 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 44; 2022: 56; 2023: 59; 2024: 51; 2025: 57; 2026: 56.

323 papers.

Sort:

By area:

2023Data Selection for Language Models via Importance Resamplingcited by 322
2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
2024Rho-1: Not All Tokens Are What You Needcited by 132
2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 114
2021Efficient training of physics-informed neural networks via importance samplingcited by 109
2025Self-Evolving Curriculum for LLM Reasoningcited by 107
2024How to Train Data-Efficient LLMscited by 91
2026TIP: Token Importance in On-Policy Distillationcited by 90
2021Chaos as an interpretable benchmark for forecasting and data-driven modellingcited by 87
2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
2021Learning Fast Sample Re-weighting Without Reward Datacited by 69

38 more

2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
2021Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inferencecited by 66
2022Client Selection in Federated Learning: Principles, Challenges, and Opportunitiescited by 62
2022Adaptive Second Order Coresets for Data-efficient Machine Learningcited by 60
2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 52
2022FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clientscited by 46
2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
2022Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attackscited by 37
2023A Survey on Efficient Training of Transformerscited by 37
2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
2022The Effect of Task Ordering in Continual Learningcited by 30
2022Not All Poisons are Created Equal: Robust Training against Data Poisoningcited by 29
2023Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understandingcited by 28
2025A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advancescited by 28
2024Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learningcited by 27
2022Rank-based Decomposable Losses in Machine Learning: A Surveycited by 26
2022Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamicscited by 26
2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
2023Teaching Language Models to Self-Improve through Interactive Demonstrationscited by 26
2023On Diversified Preferences of Large Language Model Alignmentcited by 26
2026PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competencecited by 26
2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
2024Swift Sampler: Efficient Learning of Sampler by 10 Parameterscited by 25
2022A Lagrangian Duality Approach to Active Learningcited by 24
2022DELTA: Diverse Client Sampling for Fasting Federated Learningcited by 24
2022Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Trainingcited by 24
2023Memorization Through the Lens of Curvature of Loss Function Around Samplescited by 24
2023To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasetscited by 24
2023Green Edge AI: A Contemporary Surveycited by 24
2021The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learningcited by 23
2021Interpretability-Driven Sample Selection Using Self Supervised Learning For Disease Classification And Segmentationcited by 22
2021Taxonomy of Machine Learning Safety: A Survey and Primercited by 22
2023Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Leastcited by 22

50 most cited of 323 All 323 (JSON)