mlsanityData to 14 Oct 2026

The Prevalence of Neural Collapse in Neural Multivariate Regression

arXiv 2409.04180 ↗First posted 6 Sep 2024cs.LG cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Recently it has been observed that neural networks exhibit Neural Collapse (NC) during the final stage of training for the classification problem. We empirically show that multivariate regression, as employed in imitation learning and other applications, exhibits Neural Regression Collapse (NRC), a new form of neural collapse: (NRC1) The last-layer feature vectors collapse to the subspace spanned by the n principal components of the feature vectors, where n is the dimension of the targets (for univariate regression, n=1); (NRC2) The last-layer feature vectors also collapse to the subspace spanned by the last-layer weight vectors; (NRC3) The Gram matrix for the weight vectors converges to a specific functional form that depends on the covariance matrix of the targets. After empirically establishing the prevalence of (NRC1)-(NRC3) for a variety of datasets and network architectures, we provide an explanation of these phenomena by modeling the regression task in the context of the Unconstrained Feature Model (UFM), in which the last layer feature vectors are treated as free variables when minimizing the loss function. We show that when the regularization parameters in the UFM model are strictly positive, then (NRC1)-(NRC3) also emerge as solutions in the UFM optimization problem. We also show that if the regularization parameters are equal to zero, then there is no collapse. To our knowledge, this is the first empirical and theoretical study of neural collapse in the context of regression. This extension is significant not only because it broadens the applicability of neural collapse to a new category of problems but also because it suggests that the phenomena of neural collapse could be a universal behavior in deep learning.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 6 Sep 2024.

Builds on

32 of 39 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  4. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  5. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  6. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  7. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 429
  8. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  9. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  10. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  11. 2020Neural collapse with unconstrained featurescited by 142
  12. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
20 more
  1. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  2. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  3. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  4. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  5. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  6. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  7. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  8. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  9. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  10. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 52
  11. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  12. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  13. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  14. 2020Explicit regularization and implicit bias in deep network classifiers trained with the square losscited by 37
  15. 2022A Law of Data Separation in Deep Learningcited by 36
  16. 2022Perturbation Analysis of Neural Collapsecited by 35
  17. 2024Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agentcited by 21
  18. 2024Learning Equi-angular Representations for Online Continual Learningcited by 15
  19. 2024Cross Entropy versus Label Smoothing: A Neural Collapse Perspectivecited by 11
  20. 2024Pushing Boundaries: Mixup's Influence on Neural Collapsecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 10.

21 papers.

Sort:

  1. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
  2. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  3. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 12
  4. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 10
  5. 2025Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Modelcited by 6
  6. 2026Uncovering the Latent Potential of Deep Intermediate Representationscited by 3
  7. 2025Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Modelcited by 2
  8. 2025RetroMotion: Retrocausal Motion Forecasting Models are Instructablecited by 2
  9. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  10. 2025KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approachescited by 1
  11. 2025Geometric Analysis of Neural Regression Collapse via Intrinsic Dimensioncited by 1
  12. 2026Deep Neural Regression Collapsecited by 1
9 more
  1. 2025Sharper Convergence Rates for Nonconvex Optimisation via Reduction Mappingscited by 0
  2. 2026Information Theoretic Perspective on Representation Learningcited by 0
  3. 2026Emergent Low-Rank Training Dynamics in MLPs with Smooth Activationscited by 0
  4. 2026DrawMotion: Generating 3D Human Motions by Freehand Drawingcited by 0
  5. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  6. 2026The role of class encoding in neural collapsecited by 0
  7. 2026Contrastive-Collapsed Loss for Flexible and Geometrically Optimal Embeddings and Faster Convergencecited by 0
  8. 2026One Sequence, Many Decodings: CAGenMol-2 Recasts Drug Design as Masked Molecular Inferencecited by 0
  9. 2026Understanding Head Geometry and Dynamics in Federated Regression through a Natural Solution Selection Rule: An Unconstrained Feature Model Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Model Collapse Demystified: The Case of Regression
  2. 2025Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regression
  3. 2024Strong Model Collapse
  4. 2026Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training
  5. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective
  6. 2025When Models Don't Collapse: On the Consistency of Iterative MLE
  7. 2026Dichotomy of Feature Learning and Unlearning: Fast-Slow Analysis on Neural Networks with Stochastic Gradient Descent
  8. 2024Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training