# On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Data

arXiv: https://arxiv.org/abs/2303.07608 · HTML: https://mlsanity.com/paper/2303.07608/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 43 references that are in the map.
- First posted on arXiv 14 Mar 2023.

## Builds on (most cited first, 30 of 34)

- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Gradient Descent Maximizes the Margin of Homogeneous Neural Networks](https://mlsanity.com/paper/1906.05890/) (2019)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [What is the Effect of Importance Weighting in Deep Learning?](https://mlsanity.com/paper/1812.03372/) (2018)
- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Directional convergence and alignment in deep learning](https://mlsanity.com/paper/2006.06657/) (2020)
- [Cost Sensitive Learning of Deep Feature Representations from Imbalanced Data](https://mlsanity.com/paper/1508.03422/) (2015)
- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path](https://mlsanity.com/paper/2106.02073/) (2021)
- [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020)
- [Extended Unconstrained Features Model for Exploring Deep Neural Collapse](https://mlsanity.com/paper/2202.08087/) (2022)
- [On the Role of Neural Collapse in Transfer Learning](https://mlsanity.com/paper/2112.15121/) (2021)
- [On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features](https://mlsanity.com/paper/2203.01238/) (2022)
- [An Unconstrained Layer-Peeled Perspective on Neural Collapse](https://mlsanity.com/paper/2110.02796/) (2021)
- [Risk and parameter convergence of logistic regression](https://mlsanity.com/paper/1803.07300/) (2018)
- [Imbalance Trouble: Revisiting Neural-Collapse Geometry](https://mlsanity.com/paper/2208.05512/) (2022)
- [Are All Losses Created Equal: A Neural Collapse Perspective](https://mlsanity.com/paper/2210.02192/) (2022)
- [Identifying and Compensating for Feature Deviation in Imbalanced Deep Learning](https://mlsanity.com/paper/2001.01385/) (2020)
- [Label-Imbalanced and Group-Sensitive Classification under Overparameterization](https://mlsanity.com/paper/2103.01550/) (2021)
- [Limitations of Neural Collapse for Understanding Generalization in Deep Learning](https://mlsanity.com/paper/2202.08384/) (2022)
- [Adjusting Decision Boundary for Class Imbalanced Learning](https://mlsanity.com/paper/1912.01857/) (2019)
- [Neural Collapse with Cross-Entropy Loss](https://mlsanity.com/paper/2012.08465/) (2020)
- [AutoBalance: Optimized Loss Functions for Imbalanced Data](https://mlsanity.com/paper/2201.01212/) (2022)
- [Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learning](https://mlsanity.com/paper/2204.08735/) (2022)
- [Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functions](https://mlsanity.com/paper/2209.15055/) (2022)

## Built on by (most cited first, 23 of 23)

- [Investigating the Catastrophic Forgetting in Multimodal Large Language Models](https://mlsanity.com/paper/2309.10313/) (2023)
- [Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data](https://mlsanity.com/paper/2309.09725/) (2023)
- [Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations](https://mlsanity.com/paper/2408.15417/) (2024)
- [Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model](https://mlsanity.com/paper/2401.02058/) (2024)
- [Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants](https://mlsanity.com/paper/2308.01746/) (2023)
- [Neural Collapse in Multi-label Learning with Pick-all-label Loss](https://mlsanity.com/paper/2310.15903/) (2023)
- [Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame](https://mlsanity.com/paper/2411.01248/) (2024)
- [Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data](https://mlsanity.com/paper/2409.01832/) (2024)
- [Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss](https://mlsanity.com/paper/2310.00893/) (2023)
- [Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching](https://mlsanity.com/paper/2306.07960/) (2023)
- [Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features](https://mlsanity.com/paper/2402.18884/) (2024)
- [Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics](https://mlsanity.com/paper/2512.04006/) (2025)
- [Why Loss Re-weighting Works If You Stop Early: Training Dynamics of Unconstrained Features](https://mlsanity.com/paper/2601.12011/) (2026)
- [Structure Before Collapse: Transient semantic geometry in next-token prediction](https://mlsanity.com/paper/2606.26749/) (2026)
- [Class Imbalance in Anomaly Detection: Learning from an Exactly Solvable Model](https://mlsanity.com/paper/2501.11638/) (2025)
- [Reducing Class-Wise Performance Disparity via Margin Regularization](https://mlsanity.com/paper/2602.00205/) (2026)
- [Optimizer choice matters for the emergence of Neural Collapse](https://mlsanity.com/paper/2602.16642/) (2026)
- [Optimizing for ROC Curves on Class-Imbalanced Data by Training over a Family of Loss Functions](https://mlsanity.com/paper/2402.05400/) (2024)
- [Training Over a Distribution of Hyperparameters for Enhanced Performance and Adaptability on Imbalanced Classification](https://mlsanity.com/paper/2410.03588/) (2024)
- [Enhancing Pre-Trained Model-Based Class-Incremental Learning through Neural Collapse](https://mlsanity.com/paper/2504.18437/) (2025)
- [How Label Imbalance Shapes Geometry: A General Spectral Analysis of Multi-Label Neural Collapse](https://mlsanity.com/paper/2605.01897/) (2026)
- [The Implicit Bias of Depth: From Neural Collapse to Softmax Codes](https://mlsanity.com/paper/2605.23087/) (2026)
- [Simplifying Neural Networks During Training](https://mlsanity.com/paper/2607.27854/) (2026)

## Abstract (arXiv)

Various logit-adjusted parameterizations of the cross-entropy (CE) loss have been proposed as alternatives to weighted CE for training large models on label-imbalanced data far beyond the zero train error regime. The driving force behind those designs has been the theory of implicit bias, which for linear(ized) models, explains why they successfully induce bias on the optimization path towards solutions that favor minorities. Aiming to extend this theory to non-linear models, we investigate the implicit geometry of classifiers and embeddings that are learned by different CE parameterizations. Our main result characterizes the global minimizers of a non-convex cost-sensitive SVM classifier for the unconstrained features model, which serves as an abstraction of deep nets. We derive closed-form formulas for the angles and norms of classifiers and embeddings as a function of the number of classes, the imbalance and the minority ratios, and the loss hyperparameters. Using these, we show that logit-adjusted parameterizations can be appropriately tuned to learn symmetric geometries irrespective of the imbalance ratio. We complement our analysis with experiments and an empirical study of convergence accuracy in deep-nets.