mlsanityData to 7 Oct 2026

Understanding the role of importance weighting for deep learning

arXiv 2103.15209 ↗First posted 28 Mar 2021cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

The recent paper by Byrd & Lipton (2019), based on empirical observations, raises a major concern on the impact of importance weighting for the over-parameterized deep learning models. They observe that as long as the model can separate the training data, the impact of importance weighting diminishes as the training proceeds. Nevertheless, there lacks a rigorous characterization of this phenomenon. In this paper, we provide formal characterizations and theoretical justifications on the role of importance weighting with respect to the implicit bias of gradient descent and margin-based learning theory. We reveal both the optimization dynamics and generalization performance under deep learning models. Our work not only explains the various novel phenomenons observed for importance weighting in deep learning, but also extends to the studies where the weights are being optimized as part of the model, which applies to a number of topics under active research.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 28 Mar 2021.

Builds on

33 of 47 references.

References · 33

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  4. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  5. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  6. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  8. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  9. 2014Convex Optimization: Algorithms and Complexitycited by 569
  10. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  11. 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 364
  12. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
21 more
  1. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  2. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  3. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  4. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  5. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  6. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  7. 2018What is the Effect of Importance Weighting in Deep Learning?cited by 250
  8. 2016Recommendations as Treatments: Debiasing Learning and Evaluationcited by 246
  9. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  10. 2017Learning from Noisy Labels with Distillationcited by 160
  11. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  12. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  13. 2019Regularized Learning for Domain Adaptation under Label Shiftscited by 118
  14. 2018Gradient Harmonized Single-stage Detectorcited by 114
  15. 2018Offline A/B testing for Recommender Systemscited by 94
  16. 2020Rethinking Importance Weighting for Deep Learning under Distribution Shiftcited by 92
  17. 2018Deep Imbalanced Learning for Face Recognition and Attribute Predictioncited by 90
  18. 2018Risk and parameter convergence of logistic regressioncited by 81
  19. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  20. 2015Counterfactual Risk Minimization: Learning from Logged Bandit Feedbackcited by 33
  21. 2020Adversarial Counterfactual Learning and Evaluation for Recommender Systemcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 10; 2023: 12; 2024: 4; 2025: 5; 2026: 1.

38 papers.

Sort:

By area:

  1. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  2. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 122
  3. 2022Balanced MSE for Imbalanced Visual Regressioncited by 89
  4. 2023Robust Learning with Progressive Data Expansion Against Spurious Correlationcited by 39
  5. 2022UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixupcited by 34
  6. 2022Avoiding spurious correlations via logit correctioncited by 32
  7. 2023RLSbench: Domain Adaptation Under Relaxed Label Shiftcited by 30
  8. 2022Understanding Why Generalized Reweighting Does Not Improve Over ERMcited by 29
  9. 2021SurvITE: Learning Heterogeneous Treatment Effects from Time-to-Event Datacited by 25
  10. 2023Jointly Learning Consistent Causal Abstractions Over Multiple Interventional Distributionscited by 21
  11. 2021Is Importance Weighting Incompatible with Interpolating Classifiers?cited by 19
  12. 2024A Short Survey on Importance Weighting for Machine Learningcited by 15
26 more
  1. 2021Imbalanced Adversarial Training with Reweightingcited by 11
  2. 2023When is Importance Weighting Correction Needed for Covariate Shift Adaptation?cited by 11
  3. 2023Understanding Difficulty-based Sample Weighting with a Universal Difficulty Measurecited by 9
  4. 2024Posterior Uncertainty Quantification in Neural Networks using Data Augmentationcited by 9
  5. 2021Rethinking Neural vs. Matrix-Factorization Collaborative Filtering: the Theoretical Perspectivescited by 8
  6. 2022Importance Tempering: Group Robustness for Overparameterized Modelscited by 6
  7. 2022Learning Sample Reweighting for Accuracy and Adversarial Robustnesscited by 6
  8. 2023Sample-Level Weighting for Multi-Task Learning with Auxiliary Taskscited by 6
  9. 2023Data Optimization in Deep Learning: A Surveycited by 6
  10. 2021DIVA: Dataset Derivative of a Learning Taskcited by 4
  11. 2022Exploring the Learning Difficulty of Data Theory and Measurecited by 4
  12. 2022Improving group robustness under noisy labels using predictive uncertaintycited by 4
  13. 2022Undersampling is a Minimax Optimal Robustness Intervention in Nonparametric Classificationcited by 3
  14. 2024Optimizing importance weighting in the presence of sub-population shiftscited by 3
  15. 2025Spectral Algorithms in Misspecified Regression: Convergence under Covariate Shiftcited by 3
  16. 2022Imbalanced Classification in Medical Imaging via Regroupingcited by 2
  17. 2023Pushing the Accuracy-Group Robustness Frontier with Introspective Self-playcited by 2
  18. 2023Correcting for Selection Bias and Missing Response in Regression using Privileged Informationcited by 2
  19. 2023Reweighted Mixup for Subpopulation Shiftcited by 2
  20. 2025Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regimecited by 2
  21. 2025Thumb on the Scale: Optimal Loss Weighting in Last Layer Retrainingcited by 1
  22. 2025Transferable Parasitic Estimation via Graph Contrastive Learning and Label Rebalancing in AMS Circuitscited by 1
  23. 2026Why Loss Re-weighting Works If You Stop Early: Training Dynamics of Unconstrained Featurescited by 1
  24. 2023Rescuing referral failures during automated diagnosis of domain-shifted medical imagescited by 0
  25. 2024Streamflow Prediction with Uncertainty Quantification for Water Management: A Constrained Reasoning and Learning Approachcited by 0
  26. 2025LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Trainingcited by 0