mlsanityData to 14 Oct 2026

Unique Hard Attention: A Tale of Two Sides

arXiv 2503.14615 ↗First posted 18 Mar 2025cs.LG cs.CC cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Understanding the expressive power of transformers has recently attracted attention, as it offers insights into their abilities and limitations. Many studies analyze unique hard attention transformers, where attention selects a single position that maximizes the attention scores. When multiple positions achieve the maximum score, either the rightmost or the leftmost of those is chosen. In this paper, we highlight the importance of this seeming triviality. Recently, finite-precision transformers with both leftmost- and rightmost-hard attention were shown to be equivalent to Linear Temporal Logic (LTL). We show that this no longer holds with only leftmost-hard attention -- in that case, they correspond to a strictly weaker fragment of LTL. Furthermore, we show that models with leftmost-hard attention are equivalent to soft attention, suggesting they may better approximate real-world transformers than right-attention models. These findings refine the landscape of transformer expressivity and underscore the role of attention directionality.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 18 of its 34 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

18 of 34 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  3. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  4. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 139
  5. 2023What Formal Languages Can Transformers Express? A Surveycited by 124
  6. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  7. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  8. 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 93
  9. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 75
  10. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  11. 2022A Logic for Expressing Log-Precision Transformerscited by 67
  12. 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 66
6 more
  1. 2023Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languagescited by 49
  2. 2023Logical Languages Accepted by Transformer Encoders with Hard Attentioncited by 38
  3. 2024Transformers Can Represent n-gram Language Modelscited by 32
  4. 2023Average-Hard Attention Transformers are Constant-Depth Uniform Threshold Circuitscited by 26
  5. 2024On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoningcited by 26
  6. 2024Simulating Hard Attention Using Soft Attentioncited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025Quantitative Bounds for Length Generalization in Transformerscited by 8
  2. 2025On the Reasoning Abilities of Masked Diffusion Language Modelscited by 7
  3. 2025Benefits and Limitations of Communication in Multi-Agent Reasoningcited by 7
  4. 2025Transformers are Inherently Succinctcited by 4
  5. 2025Expressive Power of Graph Transformers via Logiccited by 2
  6. 2025Probability Distributions Computed by Autoregressive Transformerscited by 2
  7. 2026Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don'tcited by 1
  8. 2026Disentangling the Expressivity of RoPEcited by 1
  9. 2026On the Computational Hardness of Transformerscited by 0
  10. 2026Encoder-Decoder Transformers: Logical Characterizations and Periodicitycited by 0