mlsanityData to 14 Oct 2026

Geometric Analysis of Neural Regression Collapse via Intrinsic Dimension

arXiv 2510.01105 ↗First posted 1 Oct 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Neural multivariate regression underpins a wide range of domains, including control, robotics, and finance, yet the geometry of its learned representations remains poorly characterized. While neural collapse has been shown to benefit generalization in classification, we find that analogous collapse in regression consistently degrades performance. To explain this contrast, we analyze regression models through the lens of intrinsic dimension. Across control tasks and synthetic datasets, we estimate the intrinsic dimension of last-layer features (ID_H) and compare it with that of the regression targets (ID_Y). Collapsed models exhibit ID_H < ID_Y, leading to over-compression and poor generalization, whereas non-collapsed models typically maintain ID_H > ID_Y. For the non-collapsed models, performance with respect to ID_H depends on the data quantity and noise levels. From these observations, we identify two regimes (over-compressed and under-compressed) that determine when expanding or reducing feature dimensionality improves performance. Our results provide new geometric insights into neural regression collapse and suggest practical strategies for enhancing generalization.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 62 of its 78 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

62 of 78 references.

References · 62

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  6. 2018DeepMind Control Suitecited by 956
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  9. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  10. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  11. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 315
  12. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
38 more
  1. 2018Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionalitycited by 282
  2. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  3. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 250
  4. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  5. 2013Predicting Parameters in Deep Learningcited by 224
  6. 2023Understanding plasticity in neural networkscited by 174
  7. 2018Dimensionality-Driven Learning with Noisy Labelscited by 173
  8. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  9. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  10. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  11. 2020Neural collapse with unconstrained featurescited by 142
  12. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  13. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  14. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  15. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  16. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  17. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  18. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  19. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  20. 2024Simplifying Deep Temporal Difference Learningcited by 78
  21. 2018Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topologycited by 73
  22. 2018The Description Length of Deep Learning Modelscited by 72
  23. 2021Intrinsic Dimension, Persistent Homology and Generalization in Neural Networkscited by 70
  24. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  25. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
  26. 2020Computing the Testing Error without a Testing Setcited by 54
  27. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  28. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  29. 2024Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimensioncited by 45
  30. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  31. 2019Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural networkcited by 39
  32. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
  33. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  34. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  35. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  36. 2021Generalization bounds via distillationcited by 34
  37. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  38. 2024Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agentcited by 21

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Scaling laws for amplitude surrogatescited by 6