# On the Dynamics Under the Unhinged Loss and Beyond

arXiv: https://arxiv.org/abs/2312.07841 · HTML: https://mlsanity.com/paper/2312.07841/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 70 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 8 of 8)

- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Feature Learning in Infinite-Width Neural Networks](https://mlsanity.com/paper/2011.14522/) (2020)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [Limitations of Neural Collapse for Understanding Generalization in Deep Learning](https://mlsanity.com/paper/2202.08384/) (2022)
- [Explicit regularization and implicit bias in deep network classifiers trained with the square loss](https://mlsanity.com/paper/2101.00072/) (2020)
- [Maximum Class Separation as Inductive Bias in One Matrix](https://mlsanity.com/paper/2206.08704/) (2022)
- [A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalization](https://mlsanity.com/paper/2304.08914/) (2023)

## Built on by (most cited first, 3 of 3)

- [Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond](https://mlsanity.com/paper/2410.12274/) (2024)
- [Symmetrization of Loss Functions for Robust Training of Neural Networks in the Presence of Noisy Labels](https://mlsanity.com/paper/2605.20347/) (2026)
- [Smoothness-Based Derandomization of PAC-Bayes Bounds](https://mlsanity.com/paper/2606.19105/) (2026)

## Abstract (arXiv)

Recent works have studied implicit biases in deep learning, especially the behavior of last-layer features and classifier weights. However, they usually need to simplify the intermediate dynamics under gradient flow or gradient descent due to the intractability of loss functions and model architectures. In this paper, we introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze the closed-form dynamics while requiring as few simplifications or assumptions as possible. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization. Based on the layer-peeled model that views last-layer features as free optimization variables, we conduct a thorough analysis in the unconstrained, regularized, and spherical constrained cases, as well as the case where the neural tangent kernel remains invariant. To bridge the performance of the unhinged loss to that of Cross-Entropy (CE), we investigate the scenario of fixing classifier weights with a specific structure, (e.g., a simplex equiangular tight frame). Our analysis shows that these dynamics converge exponentially fast to a solution depending on the initialization of features and classifier weights. These theoretical results not only offer valuable insights, including explicit feature regularization and rescaled learning rates for enhancing practical training with the unhinged loss, but also extend their applicability to other loss functions. Finally, we empirically demonstrate these theoretical results and insights through extensive experiments.