# On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers

arXiv: https://arxiv.org/abs/2012.05420 · HTML: https://mlsanity.com/paper/2012.05420/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2020.

## Built on by (most cited first, 30 of 52)

- [The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers](https://mlsanity.com/paper/2210.06313/) (2022)
- [Extended Unconstrained Features Model for Exploring Deep Neural Collapse](https://mlsanity.com/paper/2202.08087/) (2022)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold](https://mlsanity.com/paper/2209.09211/) (2022)
- [Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model](https://mlsanity.com/paper/2305.13165/) (2023)
- [Generalized Neural Collapse for a Large Number of Classes](https://mlsanity.com/paper/2310.05351/) (2023)
- [Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data](https://mlsanity.com/paper/2309.09725/) (2023)
- [Linguistic Collapse: Neural Collapse in (Large) Language Models](https://mlsanity.com/paper/2405.17767/) (2024)
- [Perturbation Analysis of Neural Collapse](https://mlsanity.com/paper/2210.16658/) (2022)
- [Improving Self-Supervised Learning by Characterizing Idealized Representations](https://mlsanity.com/paper/2209.06235/) (2022)
- [The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks](https://mlsanity.com/paper/2210.03820/) (2022)
- [Average gradient outer product as a mechanism for deep neural collapse](https://mlsanity.com/paper/2402.13728/) (2024)
- [The Prevalence of Neural Collapse in Neural Multivariate Regression](https://mlsanity.com/paper/2409.04180/) (2024)
- [A Neural Collapse Perspective on Feature Evolution in Graph Neural Networks](https://mlsanity.com/paper/2307.01951/) (2023)
- [Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?](https://mlsanity.com/paper/2405.14468/) (2024)
- [Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse](https://mlsanity.com/paper/2410.04887/) (2024)
- [Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants](https://mlsanity.com/paper/2308.01746/) (2023)
- [Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame](https://mlsanity.com/paper/2411.01248/) (2024)
- [Progressive Feedforward Collapse of ResNet Training](https://mlsanity.com/paper/2405.00985/) (2024)
- [Cross Entropy versus Label Smoothing: A Neural Collapse Perspective](https://mlsanity.com/paper/2402.03979/) (2024)
- [Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networks](https://mlsanity.com/paper/2402.03991/) (2024)
- [Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations](https://mlsanity.com/paper/2306.17105/) (2023)
- [Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data](https://mlsanity.com/paper/2409.01832/) (2024)
- [Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers](https://mlsanity.com/paper/2505.15239/) (2025)
- [Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay](https://mlsanity.com/paper/2309.04644/) (2023)
- [On the Robustness of Neural Collapse and the Neural Collapse of Robustness](https://mlsanity.com/paper/2311.07444/) (2023)
- [Intrinsic Self-Supervision for Data Quality Audits](https://mlsanity.com/paper/2305.17048/) (2023)
- [Control-oriented Clustering of Visual Latent Representation](https://mlsanity.com/paper/2410.05063/) (2024)
- [Feature Collapse](https://mlsanity.com/paper/2305.16162/) (2023)
- [Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model](https://mlsanity.com/paper/2506.05801/) (2025)

## Abstract (arXiv)

A recent numerical study observed that neural network classifiers enjoy a large degree of symmetry in the penultimate layer. Namely, if h(x) = Af(x) +b where A is a linear map and f is the output of the penultimate layer of the network (after activation), then all data points xi, 1, dots, xi, Ni in a class Ci are mapped to a single point yi by f and the points yi are located at the vertices of a regular k-1-dimensional standard simplex in a high-dimensional Euclidean space. We explain this observation analytically in toy models for highly expressive deep neural networks. In complementary examples, we demonstrate rigorously that even the final output of the classifier h is not uniform over data samples from a class Ci if h is a shallow network (or if the deeper layers do not bring the data samples into a convenient geometric configuration).