mlsanityData to 7 Oct 2026

Nearest Class-Center Simplification through Intermediate Layers

arXiv 2201.08924 ↗First posted 21 Jan 2022cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Recent advances in theoretical Deep Learning have introduced geometric properties that occur during training, past the Interpolation Threshold -- where the training error reaches zero. We inquire into the phenomena coined Neural Collapse in the intermediate layers of the networks, and emphasize the innerworkings of Nearest Class-Center Mismatch inside the deepnet. We further show that these processes occur both in vision and language model architectures. Lastly, we propose a Stochastic Variability-Simplification Loss (SVSL) that encourages better geometrical features in intermediate layers, and improves both train metrics and generalization.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 21 Jan 2022.

Builds on

33 of 47 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2016Densely Connected Convolutional Networkscited by 6,691
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2020Supervised Contrastive Learningcited by 2,803
21 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  2. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  3. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  5. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  6. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  7. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  8. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  9. 2021Deep learning: a statistical viewpointcited by 239
  10. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  11. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  12. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  13. 2020Neural collapse with unconstrained featurescited by 140
  14. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  15. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  16. 2016Robust Large Margin Deep Neural Networkscited by 117
  17. 2018Large Margin Deep Networks for Classificationcited by 110
  18. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  19. 2020Neural Collapse with Cross-Entropy Losscited by 52
  20. 2020Explaining Deep Neural Networks using Unsupervised Clusteringcited by 9
  21. 2021Sparsity-Probe: Analysis tool for Deep Learning Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 5; 2024: 6; 2025: 1; 2026: 2.

20 papers.

Sort:

By area:

  1. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  2. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  3. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  4. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  5. 2022A Law of Data Separation in Deep Learningcited by 36
  6. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  7. 2023Reverse Engineering Self-Supervised Learningcited by 33
  8. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  9. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14
  10. 2022Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learningcited by 13
  11. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  12. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
8 more
  1. 2023Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decaycited by 9
  2. 2024Transformer Block Coupling and its Correlation with Generalization in LLMscited by 9
  3. 2023On the Robustness of Neural Collapse and the Neural Collapse of Robustnesscited by 8
  4. 2025From Compression to Expression: A Layerwise Analysis of In-Context Learningcited by 5
  5. 2023Exploring Learned Representations of Neural Networks with Principal Component Analysiscited by 1
  6. 2024Where does In-context Translation Happen in Large Language Modelscited by 1
  7. 2026Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimizationcited by 0
  8. 2026Association Restoration Test: Revealing Restorable Shortcuts after Unlearningcited by 0