mlsanityData to 14 Oct 2026

Why Loss Re-weighting Works If You Stop Early: Training Dynamics of Unconstrained Features

arXiv 2601.12011 ↗First posted 17 Jan 2026cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

The application of loss reweighting in modern deep learning presents a nuanced picture. While it fails to alter the terminal learning phase in overparameterized deep neural networks (DNNs) trained on high-dimensional datasets, empirical evidence consistently shows it offers significant benefits early in training. To transparently demonstrate and analyze this phenomenon, we introduce a small-scale model (SSM). This model is specifically designed to abstract the inherent complexities of both the DNN architecture and the input data, while maintaining key information about the structure of imbalance within its spectral components. On the one hand, the SSM reveals how vanilla empirical risk minimization preferentially learns to distinguish majority classes over minorities early in training, consequently delaying minority learning. In stark contrast, reweighting restores balanced learning dynamics, enabling the simultaneous learning of features associated with both majorities and minorities.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 38 of its 44 references that are in the map. First posted on arXiv 17 Jan 2026.

Builds on

38 of 44 references.

References · 38

  1. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  4. 2020Long-tail learning via logit adjustmentcited by 618
  5. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  6. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  7. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  8. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  9. 2018What is the Effect of Importance Weighting in Deep Learning?cited by 251
  10. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  11. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  12. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
26 more
  1. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  2. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
  3. 2020Neural collapse with unconstrained featurescited by 142
  4. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 138
  5. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  6. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  7. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  8. 2020Identifying and Compensating for Feature Deviation in Imbalanced Deep Learningcited by 76
  9. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  10. 2019Adjusting Decision Boundary for Class Imbalanced Learningcited by 52
  11. 2022AutoBalance: Optimized Loss Functions for Imbalanced Datacited by 49
  12. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  13. 2021Understanding the role of importance weighting for deep learningcited by 39
  14. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  15. 2022Understanding Why Generalized Reweighting Does Not Improve Over ERMcited by 29
  16. 2020Analytic Study of Double Descent in Binary Classification: The Impact of Losscited by 28
  17. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  18. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  19. 2021Is Importance Weighting Incompatible with Interpolating Classifiers?cited by 19
  20. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  21. 2022On how to avoid exacerbating spurious correlations when models are overparameterizedcited by 4
  22. 2024The Exploration of Neural Collapse under Imbalanced Datacited by 4
  23. 2024Theoretical Guarantees of Data Augmented Last Layer Retraining Methodscited by 3
  24. 2024Sharp analysis of out-of-distribution error for "importance-weighted" estimators in the overparameterized regimecited by 3
  25. 2025An Analytical Model for Overparameterized Learning Under Class Imbalancecited by 2
  26. 2025Thumb on the Scale: Optimal Loss Weighting in Last Layer Retrainingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Controlled Face Manipulation and Synthesis for Data Augmentationcited by 1
  2. 2026Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning
  2. 2022Learning to Weight Samples for Dynamic Early-exiting Networks
  3. 2023Is Last Layer Re-Training Truly Sufficient for Robustness to Spurious Correlations?
  4. 2024Gradient Reweighting: Towards Imbalanced Class-Incremental Learning
  5. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descent
  6. 2026Efficient Unlearning through Maximizing Relearning Convergence Delay
  7. 2026Dichotomy of Feature Learning and Unlearning: Fast-Slow Analysis on Neural Networks with Stochastic Gradient Descent
  8. 2023Resilient Binary Neural Network