mlsanityData to 7 Oct 2026

The Quarks of Attention

arXiv 2202.08371 ↗First posted 15 Feb 2022cs.LG cs.AI stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Attention plays a fundamental role in both natural and artificial intelligence systems. In deep learning, attention-based neural architectures, such as transformer architectures, are widely used to tackle problems in natural language processing and beyond. Here we investigate the fundamental building blocks of attention and their computational properties. Within the standard model of deep learning, we classify all possible fundamental building blocks of attention in terms of their source, target, and computational mechanism. We identify and study three most important mechanisms: additive activation attention, multiplicative output attention (output gating), and multiplicative synaptic attention (synaptic gating). The gating mechanisms correspond to multiplicative extensions of the standard model and are used across all current attention-based deep learning architectures. We study their functional properties and estimate the capacity of several attentional building blocks in the case of linear and polynomial threshold gates. Surprisingly, additive activation attention plays a central role in the proofs of the lower bounds. Attention mechanisms reduce the depth of certain basic circuits and leverage the power of quadratic activations without incurring their full cost.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 14 of its 33 references that are in the map. First posted on arXiv 15 Feb 2022.

Builds on

14 of 33 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  5. 2014Neural Turing Machinescited by 630
  6. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  7. 2015Attention-Based Models for Speech Recognitioncited by 400
  8. 2021Pay Attention to MLPscited by 367
  9. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  10. 2016Parameterized Machine Learning for High-Energy Physicscited by 33
  11. 2020Permutationless Many-Jet Event Reconstruction with Symmetry Preserving Attention Networkscited by 22
  12. 2020SPLASH: Learnable Activation Functions for Improving Accuracy and Adversarial Robustnesscited by 18
2 more
  1. 2021SPANet: Generalized Permutationless Set Assignment for Particle Physics using Symmetry Preserving Attentioncited by 18
  2. 2019The capacity of feedforward neural networkscited by 14

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 1; 2025: 1.

7 papers.

  1. 2023Transformers as Support Vector Machinescited by 67
  2. 2023On the Role of Attention in Prompt-tuningcited by 49
  3. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  4. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  5. 2023Interpretable Joint Event-Particle Reconstruction for Neutrino Physics at NOvA with Sparse CNNs and Transformerscited by 3
  6. 2023Inductive biases in deep learning models for weather predictioncited by 3
  7. 2025Particle Hit Clustering and Identification Using Point Set Transformers in Liquid Argon Time Projection Chamberscited by 2