mlsanityData to 14 Oct 2026

Simulating Hard Attention Using Soft Attention

arXiv 2412.09925 ↗First posted 13 Dec 2024cs.LG cs.CL cs.FLcited by 15

Not on the map. Why?

Abstract (arXiv)

We study conditions under which transformers using soft attention can simulate hard attention, that is, effectively focus all attention on a subset of positions. First, we examine several subclasses of languages recognized by hard-attention transformers, which can be defined in variants of linear temporal logic. We demonstrate how soft-attention transformers can compute formulas of these logics using unbounded positional embeddings or temperature scaling. Second, we demonstrate how temperature scaling allows softmax transformers to simulate general hard-attention transformers, using a temperature that depends on the minimum gap between the maximum attention scores and other attention scores.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

22 of 33 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Layer Normalizationcited by 5,669
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  5. 2022Transformers Learn Shortcuts to Automatacited by 268
  6. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  7. 2024The Illusion of State in State-Space Modelscited by 187
  8. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  9. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  10. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  11. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  12. 2015Representation Benefits of Deep Feedforward Networkscited by 86
10 more
  1. 2024Transformers, parallel computation, and logarithmic depthcited by 79
  2. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 75
  3. 2017DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understandingcited by 71
  4. 2023Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languagescited by 49
  5. 2023Learning Transformer Programscited by 41
  6. 2024Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformerscited by 39
  7. 2023Logical Languages Accepted by Transformer Encoders with Hard Attentioncited by 38
  8. 2024Transformers Can Represent n-gram Language Modelscited by 32
  9. 2024Transformers as Transducerscited by 8
  10. 2024Improving Rare Word Translation With Dictionaries and Attention Maskingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025Unique Hard Attention: A Tale of Two Sidescited by 10
  2. 2025The Counting Power of Transformerscited by 9
  3. 2025Softmax Transformers are Turing-Completecited by 9
  4. 2025The Transformer Cookbookcited by 6
  5. 2025SlideBot: A Multi-Agent Framework for Generating Informative, Reliable, Multi-Modal Presentationscited by 5
  6. 2026Parity, Sensitivity, and Transformerscited by 5
  7. 2026Length Generalization Bounds for Transformerscited by 4
  8. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  9. 2025Concise One-Layer Transformers Can Do Function Evaluation (Sometimes)cited by 3
  10. 2026Characterizing the Expressivity of Local Attention in Transformerscited by 3
  11. 2025On the Existence of Universal Simulators of Attentioncited by 1
  12. 2025The Role of Logic and Automata in Understanding Transformerscited by 1
3 more
  1. 2026Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don'tcited by 1
  2. 2026On the Expressive Power of Transformerscited by 0
  3. 2026VC Dimension and Expressivity of Real-Valued Transformerscited by 0