mlsanityData to 7 Oct 2026

On the Unreasonable Effectiveness of Last-layer Retraining

arXiv 2512.01766 ↗First posted 1 Dec 2025cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Last-layer retraining (LLR) methods -- wherein the last layer of a neural network is reinitialized and retrained on a held-out set following ERM training -- have garnered interest as an efficient approach to rectify dependence on spurious correlations and improve performance on minority groups. Surprisingly, LLR has been found to improve worst-group accuracy even when the held-out set is an imbalanced subset of the training set. We initially hypothesize that this ``unreasonable effectiveness'' of LLR is explained by its ability to mitigate neural collapse through the held-out set, resulting in the implicit bias of gradient descent benefiting robustness. Our empirical investigation does not support this hypothesis. Instead, we present strong evidence for an alternative hypothesis: that the success of LLR is primarily due to better group balance in the held-out set. We conclude by showing how the recent algorithms CB-LLR and AFR perform implicit group-balancing to elicit a robustness improvement.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 46 of its 66 references that are in the map. First posted on arXiv 1 Dec 2025.

Builds on

46 of 66 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  9. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  10. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  12. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
34 more
  1. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 671
  2. 2018Recognition in Terra Incognitacited by 655
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  5. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  6. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  7. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 374
  8. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  9. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  10. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  11. 2021Simple data balancing achieves competitive worst-group-accuracycited by 177
  12. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  13. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  14. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  15. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  16. 2019Distributionally Robust Language Modelingcited by 142
  17. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  18. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  19. 2023Simple and Fast Group Robustness by Automatic Feature Reweightingcited by 68
  20. 2022MaskTune: Mitigating Spurious Correlations by Forcing to Explorecited by 66
  21. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  22. 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
  23. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  24. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  25. 2022Why does Throwing Away Data Improve Worst-Group Error?cited by 22
  26. 2022Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noisecited by 22
  27. 2023Simplifying Neural Network Training Under Class Imbalancecited by 18
  28. 2024Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapsecited by 13
  29. 2024Robustness to Subpopulation Shift with Domain Label Noise via Regularized Annotation of Domainscited by 6
  30. 2025Group-robust Sample Reweighting for Subpopulation Shifts via Influence Functionscited by 6
  31. 2024The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlationscited by 5
  32. 2022Improved Group Robustness via Classifier Retraining on Independent Splitscited by 2
  33. 2023TLDR: Text Based Last-layer Retraining for Debiasing Image Classifierscited by 2
  34. 2024Collapsed Language Models Promote Fairnesscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Learning from Synthetic Data via Provenance-Based Input Gradient Guidancecited by 0
  2. 2026ProME: Prototype-Margin Environments with Repair-Aware Selection for Group-Robust Learningcited by 0
  3. 2026Whitening Improves Robustness to Spurious Correlations in Linear Probescited by 0