mlsanityData to 7 Oct 2026

Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra

arXiv 2008.11865 ↗First posted 27 Aug 2020cs.LG cs.CV stat.MLcited by 94

Not on the map. Why?

Abstract (arXiv)

Numerous researchers recently applied empirical spectral analysis to the study of modern deep learning classifiers. We identify and discuss an important formal class/cross-class structure and show how it lies at the origin of the many visually striking features observed in deepnet spectra, some of which were reported in recent articles, others are unveiled here for the first time. These include spectral outliers, "spikes", and small but distinct continuous distributions, "bumps", often seen beyond the edge of a "main bulk". The significance of the cross-class structure is illustrated in three ways: (i) we prove the ratio of outliers to bulk in the spectrum of the Fisher information matrix is predictive of misclassification, in the context of multinomial logistic regression; (ii) we demonstrate how, gradually with depth, a network is able to separate class-distinctive information from class variability, all while orthogonalizing the class-distinctive information; and (iii) we propose a correction to KFAC, a well-known second-order optimization algorithm for training deepnets.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 15; 2023: 14; 2024: 22; 2025: 7; 2026: 28.

94 papers.

Sort:

By area:

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  2. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  3. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  4. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  5. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  6. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  7. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  8. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  9. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  10. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  11. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  12. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
38 more
  1. 2022A Law of Data Separation in Deep Learningcited by 36
  2. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  3. 2024Does SGD really happen in tiny subspaces?cited by 35
  4. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  5. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  6. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  7. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 27
  8. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  9. 2023ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimizationcited by 24
  10. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 20
  11. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  12. 2023Conditional Mutual Information Constrained Deep Learning for Classificationcited by 18
  13. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  14. 2022Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learningcited by 13
  15. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  16. 2024Remaining-data-free Machine Unlearning by Suppressing Sample Contributioncited by 13
  17. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  18. 2021Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stoppingcited by 12
  19. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  20. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
  21. 2024Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimizationcited by 12
  22. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  23. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  24. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  25. 2024Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Modelscited by 11
  26. 2026Insights on Muon from Simple Quadraticscited by 11
  27. 2023The Hessian perspective into the Nature of Convolutional Neural Networkscited by 10
  28. 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 10
  29. 2024Trade-Offs of Diagonal Fisher Information Matrix Estimatorscited by 10
  30. 2024Transformer Block Coupling and its Correlation with Generalization in LLMscited by 9
  31. 2024What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysiscited by 9
  32. 2026GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometrycited by 7
  33. 2023Low-Dimensional Gradient Helps Out-of-Distribution Detectioncited by 6
  34. 2026A Defense of the Quadratic Modelcited by 6
  35. 2023Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networkscited by 5
  36. 2023Unveiling the Hessian's Connection to the Decision Boundarycited by 5
  37. 2022Learning Representation from Neural Fisher Kernel with Low-rank Approximationcited by 4
  38. 2024Deconstructing the Goldilocks Zone of Neural Network Initializationcited by 4

50 most cited of 94 All 94 (JSON)