mlsanityData to 7 Oct 2026

Reorganizing attention-space geometry with expressive attention

arXiv 2407.18601 ↗First posted 26 Jul 2024cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Attention regulates information transfer between tokens. For this, query and key vectors are compared, typically in terms of a scalar product, QTK, together with a subsequent softmax normalization. In geometric terms, the standard dot-product attention (DPA) leads to large/small attention weights for parallel/antiparallel queries and keys. Here we study expressive attention (EA), which is based on (QTK)2, the squared dot product. In this case, attention is enhanced when query and key are either parallel or antiparallel, and suppressed for orthogonal configurations. EA can be introduced into any attention-based code without additional compute costs or memory requirements. For a series of autoregressive prediction tasks, we find that expressive attention performs at least as well as vanilla DPA. Increasing task complexity, EA is observed to outperform DPA with increasing margins, which also holds for multi-task settings. For a given model size, EA manages to achieve 100% performance for a range of complexity levels not accessible to DPA. Our results show that it is possible to reorganize the geometry of the matching condition in the space of attention heads without loss of performance.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 40 references that are in the map. First posted on arXiv 26 Jul 2024.

Builds on

35 of 40 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  11. 2024Large Language Models: A Surveycited by 506
  12. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
23 more
  1. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  2. 2021Random Feature Attentioncited by 350
  3. 2014The Loss Surfaces of Multilayer Networkscited by 278
  4. 2023Challenges and Applications of Large Language Modelscited by 251
  5. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  6. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  7. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  8. 2023The Quantization Model of Neural Scalingcited by 137
  9. 2023What Formal Languages Can Transformers Express? A Surveycited by 120
  10. 2022The Devil in Linear Transformercited by 98
  11. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  12. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  13. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  14. 2022Flowformer: Linearizing Transformers with Conservation Flowscited by 69
  15. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  16. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 37
  17. 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
  18. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  19. 2023Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representationcited by 28
  20. 2024LLMs for Relational Reasoning: How Far are We?cited by 21
  21. 2024Attention as an RNNcited by 6
  22. 2023Are LLMs Rigorous Logical Reasoners? Empowering Natural Language Proof Generation by Stepwise Decoding with Contrastive Learningcited by 2
  23. 2021Local homeostatic regulation of the spectral radius of echo-state networkscited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Small transformer architectures for task switchingcited by 1