mlsanityData to 7 Oct 2026

On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers

arXiv 2012.05420 ↗First posted 10 Dec 2020cs.LG stat.MLcited by 51

Not on the map. Why?

Abstract (arXiv)

A recent numerical study observed that neural network classifiers enjoy a large degree of symmetry in the penultimate layer. Namely, if h(x) = Af(x) +b where A is a linear map and f is the output of the penultimate layer of the network (after activation), then all data points xi, 1, dots, xi, Ni in a class Ci are mapped to a single point yi by f and the points yi are located at the vertices of a regular k-1-dimensional standard simplex in a high-dimensional Euclidean space. We explain this observation analytically in toy models for highly expressive deep neural networks. In complementary examples, we demonstrate rigorously that even the final output of the classifier h is not uniform over data samples from a class Ci if h is a shallow network (or if the deeper layers do not bring the data samples into a convenient geometric configuration).

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 13; 2024: 14; 2025: 8; 2026: 6.

51 papers.

Sort:

By area:

  1. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  2. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  3. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  4. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  5. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  6. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  7. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  8. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  9. 2022Perturbation Analysis of Neural Collapsecited by 34
  10. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  11. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  12. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 21
38 more
  1. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  2. 2023A Neural Collapse Perspective on Feature Evolution in Graph Neural Networkscited by 19
  3. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  4. 2023Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variantscited by 14
  5. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14
  6. 2024Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Framecited by 14
  7. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
  8. 2024Cross Entropy versus Label Smoothing: A Neural Collapse Perspectivecited by 11
  9. 2023Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representationscited by 10
  10. 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 10
  11. 2023Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decaycited by 9
  12. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 9
  13. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 9
  14. 2023Intrinsic Self-Supervision for Data Quality Auditscited by 8
  15. 2023On the Robustness of Neural Collapse and the Neural Collapse of Robustnesscited by 8
  16. 2024Control-oriented Clustering of Visual Latent Representationcited by 8
  17. 2023Feature Collapsecited by 6
  18. 2025Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Modelcited by 6
  19. 2022Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionalitycited by 4
  20. 2022FG-UAP: Feature-Gathering Universal Adversarial Perturbationcited by 4
  21. 2023Geometric structure of shallow neural networks and constructive mathcal L2 cost minimizationcited by 4
  22. 2024Can Kernel Methods Explain How the Data Affects Neural Collapse?cited by 4
  23. 2024Dual-Head Knowledge Distillation: Enhancing Logits Utilization with an Auxiliary Headcited by 4
  24. 2025Global Minimizers of Sigmoid Contrastive Losscited by 4
  25. 2023Minimum norm interpolation by perceptra: Explicit regularization and implicit biascited by 3
  26. 2022Memorization-Dilation: Modeling Neural Collapse Under Label Noisecited by 2
  27. 2023Understanding Prompt Tuning for V-L Models Through the Lens of Neural Collapsecited by 2
  28. 2024Gradient flow in parameter space is equivalent to linear interpolation in output spacecited by 2
  29. 2025Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representationscited by 2
  30. 2025Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Modelcited by 2
  31. 2025Neural Collapse in Test-Time Adaptationcited by 1
  32. 2025Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learningcited by 0
  33. 2025Enhancing Pre-Trained Model-Based Class-Incremental Learning through Neural Collapsecited by 0
  34. 2026Supervised sparse auto-encoders for interpretable and compositional representationscited by 0
  35. 2026Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Modelscited by 0
  36. 2026The role of class encoding in neural collapsecited by 0
  37. 2026Optical-Guided Neural Collapse for SAR Few-Shot Class Incremental Learningcited by 0
  38. 2026Neural-Collapse-guided Task-Free Continual Anomaly Detectioncited by 0

50 most cited of 51 All 51 (JSON)