Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining
Not on the map. Why?
Abstract (arXiv)
While machine learning models become more capable in discriminative tasks at scale, their ability to overcome biases introduced by training data has come under increasing scrutiny. Previous results suggest that there are two extremes of parameterization with very different behaviors: the population (underparameterized) setting where loss weighting is optimal and the separable overparameterized setting where loss weighting is ineffective at ensuring equal performance across classes. This work explores the regime of last layer retraining (LLR) in which the unseen limited (retraining) data is frequently inseparable and the model proportionately sized, falling between the two aforementioned extremes. We show, in theory and practice, that loss weighting is still effective in this regime, but that these weights must take into account the relative overparameterization of the model.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 24 Jun 2025.
Builds on
23 of 32 references.
References · 23
- 2014Deep Learning Face Attributes in the Wildcited by 3,693
- 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2020Long-tail learning via logit adjustmentcited by 618
- 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
- 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
- 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
- 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
- 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 138
- 2021Towards a Theoretical Framework of Out-of-Distribution Generalizationcited by 94
- 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
- 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
11 more
- 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 54
- 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
- 2013A framework to characterize performance of LASSO algorithmscited by 40
- 2021Understanding the role of importance weighting for deep learningcited by 39
- 2020LQF: Linear Quadratic Fine-Tuningcited by 25
- 2022Why does Throwing Away Data Improve Worst-Group Error?cited by 22
- 2020Sharp Asymptotics and Optimal Performance for Inference in Binary Modelscited by 13
- 2022On how to avoid exacerbating spurious correlations when models are overparameterizedcited by 4
- 2024Theoretical Guarantees of Data Augmented Last Layer Retraining Methodscited by 3
- 2024Sharp analysis of out-of-distribution error for "importance-weighted" estimators in the overparameterized regimecited by 3
- 2025A statistical theory of overfitting for imbalanced classificationcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.