mlsanityData to 7 Oct 2026

Feature Collapse

arXiv 2305.16162 ↗First posted 25 May 2023cs.LG cs.AI cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

We formalize and study a phenomenon called feature collapse that makes precise the intuitive idea that entities playing a similar role in a learning task receive similar representations. As feature collapse requires a notion of task, we leverage a simple but prototypical NLP task to study it. We start by showing experimentally that feature collapse goes hand in hand with generalization. We then prove that, in the large sample limit, distinct words that play identical roles in this NLP task receive identical local feature representations in a neural network. This analysis reveals the crucial role that normalization mechanisms, such as LayerNorm, play in feature collapse and in generalization.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 13 of its 16 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

13 of 16 references.

References · 13

  1. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  2. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  3. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  4. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  5. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  6. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  7. 2020Neural collapse with unconstrained featurescited by 140
  8. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  9. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  10. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  11. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
  12. 2020Neural Collapse with Cross-Entropy Losscited by 52
1 more
  1. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 3; 2026: 2.

6 papers.

  1. 2025Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attentioncited by 5
  2. 2023Emergence and Function of Abstract Representations in Self-Supervised Transformerscited by 3
  3. 2026Student Guides Teacher: Weak-to-Strong Inference via Spectral Orthogonal Explorationcited by 3
  4. 2025Countering Multi-modal Representation Collapse through Rank-targeted Fusioncited by 1
  5. 2025Multi-layer Radial Basis Function Networks for Out-of-distribution Detectioncited by 0
  6. 2026Dissecting Representation Structure in Vision Transformers: A Rigorous Architectural Studycited by 0