mlsanityData to 14 Oct 2026

Optimizing importance weighting in the presence of sub-population shifts

arXiv 2410.14315 ↗First posted 18 Oct 2024stat.ML cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

A distribution shift between the training and test data can severely harm performance of machine learning models. Importance weighting addresses this issue by assigning different weights to data points during training. We argue that existing heuristics for determining the weights are suboptimal, as they neglect the increase of the variance of the estimated model due to the finite sample size of the training data. We interpret the optimal weights in terms of a bias-variance trade-off, and propose a bi-level optimization procedure in which the weights and model parameters are optimized simultaneously. We apply this optimization to existing importance weighting techniques for last-layer retraining of deep neural networks in the presence of sub-population shifts and show empirically that optimizing weights significantly improves generalization performance.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 47 of its 66 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

47 of 66 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  7. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  8. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  9. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  10. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  11. 2018Learning to Reweight Examples for Robust Deep Learningcited by 675
  12. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
35 more
  1. 2018A Reductions Approach to Fair Classificationcited by 587
  2. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  3. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  4. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  5. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  6. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  7. 2018What is the Effect of Importance Weighting in Deep Learning?cited by 251
  8. 2016Hyperparameter optimization with approximate gradientcited by 233
  9. 2021Simple data balancing achieves competitive worst-group-accuracycited by 178
  10. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 172
  11. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  12. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 153
  13. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  14. 2018Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objectscited by 143
  15. 2019Distributionally Robust Language Modelingcited by 143
  16. 2020Learning from Failure: Training Debiased Classifier from Biased Classifiercited by 133
  17. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 120
  18. 2021Do Concept Bottleneck Models Learn as Intended?cited by 116
  19. 2022Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimationcited by 88
  20. 2019Predictive Inequity in Object Detectioncited by 77
  21. 2016Places: An Image Database for Deep Scene Understandingcited by 76
  22. 2023Simple and Fast Group Robustness by Automatic Feature Reweightingcited by 68
  23. 2021Causally motivated Shortcut Removal Using Auxiliary Labelscited by 55
  24. 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
  25. 2021A Theory of Label Propagation for Subpopulation Shiftcited by 48
  26. 2022Probing Classifiers are Unreliable for Concept Removal and Detectioncited by 41
  27. 2021Understanding the role of importance weighting for deep learningcited by 39
  28. 2019Domain Aggregation Networks for Multi-Source Domain Adaptationcited by 37
  29. 2020Deep Learning Applied to Chest X-Rays: Exploiting and Preventing Shortcutscited by 29
  30. 2022Are All Spurious Features in Natural Language Alike? An Analysis through a Causal Lenscited by 28
  31. 2023Discovering environments with XRMcited by 26
  32. 2022On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlationscited by 19
  33. 2024A Short Survey on Importance Weighting for Machine Learningcited by 15
  34. 2024Improving Group Robustness on Spurious Correlation Requires Preciser Group Inferencecited by 15
  35. 2023Removing Spurious Concepts from Neural Network Representations via Joint Subspace Estimationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025A Fully First-Order Layer for Differentiable Optimizationcited by 3
  2. 2026Fix Representation (Optimally) Before Fairness: Finite-Sample Shrinkage Population Correction and the True Price of Fairness Under Subpopulation Shiftcited by 0
  3. 2026Evolution-Inspired Sample Competition for Deep Neural Network Optimizationcited by 0