Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurement
Not on the map. Why?
Abstract (arXiv)
Dynamic feature transformation (the rich regime) does not always align with predictive performance (better representation), yet accuracy is often used as a proxy for richness, limiting analysis of their relationship. We propose a computationally efficient, performance-independent metric of richness grounded in the low-rank bias of rich dynamics, which recovers neural collapse as a special case. The metric is empirically more stable than existing alternatives and captures known lazy-torich transitions (e.g., grokking) without relying on accuracy. We further use it to examine how training factors (e.g., learning rate) relate to richness, confirming recognized assumptions and highlighting new observations (e.g., batch normalization promotes rich dynamics). An eigendecomposition-based visualization is also introduced to support interpretability, together providing a diagnostic tool for studying the relationship between training factors, dynamics, and representations.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 49 of its 82 references that are in the map. First posted on arXiv 5 Oct 2024.
Builds on
49 of 82 references.
References · 49
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
37 more
- 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
- 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
- 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
- 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
- 2018Gradient descent aligns the layers of deep linear networkscited by 182
- 2018A mathematical theory of semantic development in deep neural networkscited by 182
- 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
- 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
- 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
- 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
- 2020Neural collapse with unconstrained featurescited by 140
- 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
- 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
- 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
- 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
- 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
- 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
- 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
- 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
- 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
- 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
- 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
- 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
- 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
- 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
- 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
- 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
- 2022A Law of Data Separation in Deep Learningcited by 36
- 2019Pathological spectra of the Fisher information metric and its variants in deep neural networkscited by 32
- 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
- 2024An exactly solvable model for emergence and scaling laws in the multitask sparse parity problemcited by 21
- 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
- 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
- 2023Quantifying the Variability Collapse of Neural Networkscited by 9
- 2023On the Robustness of Neural Collapse and the Neural Collapse of Robustnesscited by 8
- 2021Double-descent curves in neural networks: a new perspective using Gaussian processescited by 5
- 2022Memorization-Dilation: Modeling Neural Collapse Under Label Noisecited by 2
Built on by
Citing papers per year: 2025: 2.
2 papers.