mlsanityData to 14 Oct 2026

Geometric Capacity of Transformers: A Tropical Geometry Perspective

arXiv 2604.14727 ↗First posted 16 Apr 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

To quantify the geometric capacity of transformers, we develop a tropical-geometric framework for analyzing the spatial partitions induced by conditioned self-attention. In the zero-temperature limit, we show that fixed-key top-1 routing is exactly represented by a power diagram in query space, while an auxiliary log-lifted value parameterization yields a vector-valued tropical rational representation. For Multi-Head Self-Attention (MHSA) with sequence length N and H attention heads, the joint routing geometry is encoded by Minkowski sums of headwise Newton polytopes, giving an O(NH) universal bound that sharpens to O((HN)^dmodel-1) once the number of heads reaches the intrinsic dimension dmodel. Extending this analysis across depth L, we derive the first tight asymptotic bounds on the number of linear regions in transformers (Theta!left(N^minH,dmodel-1Lright)). We further show that finite-temperature softmax preserves the top-1 routing structure and admits exponentially decaying local approximation and differential bounds away from routing boundaries.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 43 references that are in the map. First posted on arXiv 16 Apr 2026.

Builds on

24 of 43 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  3. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  4. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  5. 2019Complexity of Linear Regions in Deep Networkscited by 167
  6. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  7. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  8. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 139
  9. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  10. 2018Tropical Geometry of Deep Neural Networkscited by 95
  11. 2020On the Number of Linear Regions of Convolutional Neural Networkscited by 49
  12. 2020On the Decision Boundaries of Neural Networks: A Tropical Geometry Perspectivecited by 25
12 more
  1. 2024Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modelingcited by 25
  2. 2024Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Surveycited by 21
  3. 2025Exact Expressive Power of Transformers with Paddingcited by 19
  4. 2023Explainability of Vision Transformers: A Comprehensive Review and New Perspectivescited by 12
  5. 2024Theoretical Constraints on the Expressive Power of mathsfRoPE-based Tensor Attention Transformerscited by 12
  6. 2025Tropical Attention: Neural Algorithmic Reasoning for Combinatorial Algorithmscited by 12
  7. 2024Transformers are Expressive, But Are They Expressive Enough for Regression?cited by 6
  8. 2024TropNNC: Structured Neural Network Compression Using Tropical Geometrycited by 3
  9. 2026Sparsity is Combinatorial Depth: Quantifying MoE Expressivity via Tropical Geometrycited by 3
  10. 2025Expressive Power of Graph Transformers via Logiccited by 2
  11. 2026The Geometry of Thought: Disclosing the Transformer as a Tropical Polynomial Circuitcited by 2
  12. 2026On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functionscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Laguerre Geometry for Interpreting Large Language Modelscited by 0