mlsanityData to 14 Oct 2026

Importance Tempering: Group Robustness for Overparameterized Models

arXiv 2209.08745 ↗First posted 19 Sep 2022cs.LG cs.AI math.STcited by 6

Not on the map. Why?

Abstract (arXiv)

Although overparameterized models have shown their success on many machine learning tasks, the accuracy could drop on the testing distribution that is different from the training one. This accuracy drop still limits applying machine learning in the wild. At the same time, importance weighting, a traditional technique to handle distribution shifts, has been demonstrated to have less or even no effect on overparameterized models both empirically and theoretically. In this paper, we propose importance tempering to improve the decision boundary and achieve consistently better results for overparameterized models. Theoretically, we justify that the selection of group temperature can be different under label shift and spurious correlation setting. At the same time, we also prove that properly selected temperatures can extricate the minority collapse for imbalanced classification. Empirically, we achieve state-of-the-art results on worst group classification tasks using importance tempering.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 51 of its 67 references that are in the map. First posted on arXiv 19 Sep 2022.

Builds on

51 of 67 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2016Wide Residual Networkscited by 3,070
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  7. 2019Invariant Risk Minimizationcited by 1,854
  8. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  9. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  10. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  11. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  12. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
38 more
  1. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  2. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  4. 2020Long-tail learning via logit adjustmentcited by 618
  5. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  6. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 513
  7. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  8. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
  9. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
  10. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  11. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  12. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  13. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  14. 2018What is the Effect of Importance Weighting in Deep Learning?cited by 251
  15. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  16. 2020Directional convergence and alignment in deep learningcited by 178
  17. 2021Simple data balancing achieves competitive worst-group-accuracycited by 178
  18. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  19. 2020Seesaw Loss for Long-Tailed Instance Segmentationcited by 146
  20. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  21. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  22. 2018Stochastic subgradient method converges on tame functionscited by 124
  23. 2020A Unified View of Label Shift Estimationcited by 119
  24. 2018Convergence of Gradient Descent on Separable Datacited by 110
  25. 2020Rethinking Importance Weighting for Deep Learning under Distribution Shiftcited by 93
  26. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  27. 2019Striking the Right Balance with Uncertaintycited by 79
  28. 2020Identifying and Compensating for Feature Deviation in Imbalanced Deep Learningcited by 76
  29. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  30. 2020Gradient descent follows the regularization path for general lossescited by 59
  31. 2019The Barron Space and the Flow-induced Function Spaces for Neural Network Modelscited by 39
  32. 2021Understanding the role of importance weighting for deep learningcited by 39
  33. 2022Understanding Why Generalized Reweighting Does Not Improve Over ERMcited by 29
  34. 2021Is Importance Weighting Incompatible with Interpolating Classifiers?cited by 19
  35. 2021Training Over-parameterized Models with Non-decomposable Objectivescited by 11
  36. 2017One-shot Face Recognition by Promoting Underrepresented Classescited by 8
  37. 2021Stochastic Reweighted Gradient Descentcited by 7
  38. 2022Learning Towards the Largest Marginscited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 2.

6 papers.

  1. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  2. 2024A Short Survey on Importance Weighting for Machine Learningcited by 15
  3. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  4. 2023Bitrate-Constrained DRO: Beyond Worst Case Robustness To Unknown Group Shiftscited by 9
  5. 2022Malign Overfitting: Interpolation Can Provably Preclude Invariancecited by 6
  6. 2024Evaluating the Fairness of Neural Collapse in Medical Image Classificationcited by 1