mlsanityData to 14 Oct 2026

Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention

arXiv 2608.30417 ↗First posted 31 Aug 2026cs.LG math.AG math.RTcited by 0

Not on the map. Why?

Abstract (arXiv)

We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group G, then G can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action. As a consequence, we prove that any fixed MHSA architecture that achieves exact equivariance by polynomially parameterizing unconstrained MHSA parameters inevitably leads to expressivity loss within the class of equivariant maps: the equivariance locus of unconstrained MHSA forms a union of extremely many Zariski-irreducible components in a reduced parameter space, and any single architecture covers at most one. For G=D4 acting on C copies of the regular representation as the token feature space, we show that there are Omega(C64) components for eight attention heads. We illustrate this with experiments on artificial tasks and show that the predicted head-cluster structures can arise naturally when the attention layer is pressured to become equivariant.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

26 of 33 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2016Group Equivariant Convolutional Networkscited by 1,165
  5. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  6. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 607
  7. 2019General E(2)-Equivariant Steerable CNNscited by 446
  8. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 354
  9. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 300
  10. 2020Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Datacited by 248
  11. 2018A General Theory of Equivariant CNNs on Homogeneous Spacescited by 228
  12. 2018Universal approximations of invariant maps by neural networkscited by 127
14 more
  1. 2020LieTransformer: Equivariant self-attention for Lie Groupscited by 100
  2. 2018Roto-Translation Covariant Convolutional Networks for Medical Image Analysiscited by 92
  3. 2021Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifoldscited by 73
  4. 2020Group Equivariant Stand-Alone Self-Attention For Visioncited by 63
  5. 2020Universal Equivariant Multilayer Perceptronscited by 49
  6. 2023E(2)-Equivariant Vision Transformercited by 27
  7. 2023Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networkscited by 23
  8. 2024Geometry of Lightning Self-Attention: Identifiability and Dimensioncited by 16
  9. 2026Identifiable Equivariant Networks are Layerwise Equivariantcited by 7
  10. 2023Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groupscited by 5
  11. 2025Quick ViTs: Speeding up Vision Transformers through Equivariancecited by 3
  12. 2025On Universality of Deep Equivariant Networkscited by 3
  13. 2026Vanilla Group Equivariant Vision Transformer: Simple and Effectivecited by 3
  14. 2026Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformerscited by 1

Built on by

Not cited yet.