# Normalized Attention Without Probability Cage

arXiv: https://arxiv.org/abs/2005.09561 · HTML: https://mlsanity.com/paper/2005.09561/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2020.

## Built on by (most cited first, 20 of 20)

- [Combinatorial optimization and reasoning with graph neural networks](https://mlsanity.com/paper/2102.09544/) (2021)
- [The CLRS Algorithmic Reasoning Benchmark](https://mlsanity.com/paper/2205.15659/) (2022)
- [Cramming: Training a Language Model on a Single GPU in One Day](https://mlsanity.com/paper/2212.14034/) (2022)
- [Graph Neural Networks are Dynamic Programmers](https://mlsanity.com/paper/2203.15544/) (2022)
- [Encryption-Friendly LLM Architecture](https://mlsanity.com/paper/2410.02486/) (2024)
- [Which transformer architecture fits my data? A vocabulary bottleneck in self-attention](https://mlsanity.com/paper/2105.03928/) (2021)
- [Learnable Commutative Monoids for Graph Neural Networks](https://mlsanity.com/paper/2212.08541/) (2022)
- [Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks](https://mlsanity.com/paper/2603.11487/) (2026)
- [Breaking through the learning plateaus of in-context learning in Transformer](https://mlsanity.com/paper/2309.06054/) (2023)
- [ZeroS: Zero-Sum Linear Attention for Efficient Transformers](https://mlsanity.com/paper/2602.05230/) (2026)
- [TrimBERT: Tailoring BERT for Trade-offs](https://mlsanity.com/paper/2202.12411/) (2022)
- [HyperMLP: An Integrated Perspective for Sequence Modeling](https://mlsanity.com/paper/2602.12601/) (2026)
- [Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention](https://mlsanity.com/paper/2602.23057/) (2026)
- [Rethinking Efficacy of Softmax for Lightweight Non-Local Neural Networks](https://mlsanity.com/paper/2207.13423/) (2022)
- [Generalized Probabilistic Attention Mechanism in Transformers](https://mlsanity.com/paper/2410.15578/) (2024)
- [Design Principles for Sequence Models via Coefficient Dynamics](https://mlsanity.com/paper/2510.09389/) (2025)
- [Expressive Power and Loss Surfaces of Deep Learning Models](https://mlsanity.com/paper/2108.03579/) (2021)
- [Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2112.05343/) (2021)
- [What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation](https://mlsanity.com/paper/2607.22781/) (2026)
- [PANDA: Prior-guided Attentional Dual-path Architecture](https://mlsanity.com/paper/2610.13635/) (2026)

## Abstract (arXiv)

Attention architectures are widely used; they recently gained renewed popularity with Transformers yielding a streak of state of the art results. Yet, the geometrical implications of softmax-attention remain largely unexplored. In this work we highlight the limitations of constraining attention weights to the probability simplex and the resulting convex hull of value vectors. We show that Transformers are sequence length dependent biased towards token isolation at initialization and contrast Transformers to simple max- and sum-pooling - two strong baselines rarely reported. We propose to replace the softmax in self-attention with normalization, yielding a hyperparameter and data-bias robust, generally applicable architecture. We support our insights with empirical results from more than 25,000 trained models. All results and implementations are made available.