Deep Neural Regression Collapse
Not on the map. Why?
Abstract (arXiv)
Neural Collapse is a phenomenon that helps identify sparse and low rank structures in deep classifiers. Recent work has extended the definition of neural collapse to regression problems, albeit only measuring the phenomenon at the last layer. In this paper, we establish that Neural Regression Collapse (NRC) also occurs below the last layer across different types of models. We show that in the collapsed layers of neural regression models, features lie in a subspace that corresponds to the target dimension, the feature covariance aligns with the target covariance, the input subspace of the layer weights aligns with the feature subspace, and the linear prediction error of the features is close to the overall prediction error of the model. In addition to establishing Deep NRC, we also show that models that exhibit Deep NRC learn the intrinsic dimension of low rank targets and explore the necessity of weight decay in inducing Deep NRC. This paper provides a more complete picture of the simple structure learned by deep networks in the context of regression.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 25 Mar 2026.
Builds on
29 of 43 references.
References · 29
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
- 2017End-to-end Driving via Conditional Imitation Learningcited by 485
- 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 429
- 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
- 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
- 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
- 2020Neural collapse with unconstrained featurescited by 142
- 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 138
- 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
- 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
- 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
17 more
- 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
- 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
- 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
- 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
- 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
- 2022A Law of Data Separation in Deep Learningcited by 36
- 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
- 2023A Neural Collapse Perspective on Feature Evolution in Graph Neural Networkscited by 19
- 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
- 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 10
- 2023Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decaycited by 9
- 2023On the Robustness of Neural Collapse and the Neural Collapse of Robustnesscited by 9
- 2024Memory-efficient Continual Learning with Neural Collapse Contrastivecited by 8
- 2024On Generalization Bounds for Neural Networks with Low Rank Layerscited by 4
- 2025Generalization Bounds for Rank-sparse Neural Networkscited by 3
- 2024Spring-block theory of feature learning in deep neural networkscited by 2
- 2025Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Modelcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.