mlsanityData to 14 Oct 2026

Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model

arXiv 2506.05801 ↗First posted 6 Jun 2025cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

A phenomenon known as ''Neural Collapse (NC)'' in deep classification tasks, in which the penultimate-layer features and the final classifiers exhibit an extremely simple geometric structure, has recently attracted considerable attention, with the expectation that it can deepen our understanding of how deep neural networks behave. The Unconstrained Feature Model (UFM) has been proposed to explain NC theoretically, and there emerges a growing body of work that extends NC to tasks other than classification and leverages it for practical applications. In this study, we investigate whether a similar phenomenon arises in deep Ordinal Regression (OR) tasks, via combining the cumulative link model for OR and UFM. We show that a phenomenon we call Ordinal Neural Collapse (ONC) indeed emerges and is characterized by the following three properties: (ONC1) all optimal features in the same class collapse to their within-class mean when regularization is applied; (ONC2) these class means align with the classifier, meaning that they collapse onto a one-dimensional subspace; (ONC3) the optimal latent variables (corresponding to logits or preactivations in classification tasks) are aligned according to the class order, and in particular, in the zero-regularization limit, a highly local and simple geometric relationship emerges between the latent variables and the threshold values. We prove these properties analytically within the UFM framework with fixed threshold values and corroborate them empirically across a variety of datasets. We also discuss how these insights can be leveraged in OR, highlighting the use of fixed thresholds.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

27 of 51 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Densely Connected Convolutional Networkscited by 6,720
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  5. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 429
  6. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  7. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  8. 2020Neural collapse with unconstrained featurescited by 142
  9. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  10. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  11. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  12. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
15 more
  1. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  2. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 52
  3. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  4. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  5. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  6. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  7. 2022A Law of Data Separation in Deep Learningcited by 36
  8. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  9. 2023Reverse Engineering Self-Supervised Learningcited by 33
  10. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  11. 2023Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networkscited by 21
  12. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  13. 2024Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Modelcited by 19
  14. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
  15. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026DVD: Deterministic Video Depth Estimation with Generative Priorscited by 4
  2. 2025Geometric Analysis of Neural Regression Collapse via Intrinsic Dimensioncited by 1
  3. 2026Reducing Class-Wise Performance Disparity via Margin Regularizationcited by 1
  4. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  5. 2026Ordinal Neural Collapse as a Representation Prior for Visual Navigationcited by 0
  6. 2026Understanding Head Geometry and Dynamics in Federated Regression through a Natural Solution Selection Rule: An Unconstrained Feature Model Analysiscited by 0