mlsanityData to 7 Oct 2026

Geometry of Lightning Self-Attention: Identifiability and Dimension

arXiv 2408.17221 ↗First posted 30 Aug 2024cs.LG math.AGcited by 16

Not on the map. Why?

Abstract (arXiv)

We consider function spaces defined by self-attention networks without normalization, and theoretically analyze their geometry. Since these networks are polynomial, we rely on tools from algebraic geometry. In particular, we study the identifiability of deep attention by providing a description of the generic fibers of the parametrization for an arbitrary number of layers and, as a consequence, compute the dimension of the function space. Additionally, for a single-layer model, we characterize the singular and boundary points. Finally, we formulate a conjectural extension of our results to normalized self-attention networks, prove it for a single layer, and numerically verify it in the deep case.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 17 of its 29 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

17 of 29 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  4. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  5. 2018Music Transformercited by 207
  6. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  7. 2019On the Expressive Power of Deep Polynomial Neural Networkscited by 65
  8. 2022A General Survey on Attention Mechanisms in Deep Learningcited by 64
  9. 2024Finding Transformer Circuits with Edge Pruningcited by 58
  10. 2023Hidden symmetries of ReLU networkscited by 38
  11. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  12. 2019Pure and Spurious Critical Points: a Geometric Study of Linear Networkscited by 33
5 more
  1. 2021Geometry of Linear Convolutional Networkscited by 27
  2. 2022Symmetry Teleportation for Accelerated Optimizationcited by 25
  3. 2021Parameter identifiability of a deep feedforward ReLU neural networkcited by 24
  4. 2024Geometry of Polynomial Neural Networkscited by 21
  5. 2023Function Space and Critical Points of Linear Convolutional Networkscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 8.

16 papers.

Sort:

  1. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  2. 2025Symmetry in Neural Network Parameter Spacescited by 31
  3. 2025Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometrycited by 21
  4. 2025Identifiability of Deep Polynomial Neural Networkscited by 12
  5. 2024On the Geometry and Optimization of Polynomial Convolutional Networkscited by 11
  6. 2025Geometry and Optimization of Shallow Polynomial Networkscited by 10
  7. 2025Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networkscited by 9
  8. 2026Identifiable Equivariant Networks are Layerwise Equivariantcited by 7
  9. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  10. 2026Robustness Verification of Polynomial Neural Networkscited by 3
  11. 2025Critical Points of Degenerate Metrics on Algebraic Varieties: A Tale of Overparametrizationcited by 1
  12. 2026Algebraic Invariants of Lightning Self-Attentioncited by 1
4 more
  1. 2026Minimal Filling Architectures of Polynomial Neural Networks: Counterexamples, Frontier Search, and Defectscited by 1
  2. 2026Singular Learning and Occam's Razor in Deep Monomial Networkscited by 0
  3. 2026Linear Independence of Polynomial Compositions and Identifiability of Deep Neural Networkscited by 0
  4. 2026No Equivariant Architecture Covers All Equivariant Attentioncited by 0