# Scalable-Softmax Is Superior for Attention

arXiv: https://arxiv.org/abs/2501.19399 · HTML: https://mlsanity.com/paper/2501.19399/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2025.

## Built on by (most cited first, 30 of 45)

- [Ministral 3](https://mlsanity.com/paper/2601.08584/) (2026)
- [The Mean-Field Dynamics of Transformers](https://mlsanity.com/paper/2512.01868/) (2025)
- [A multiscale analysis of mean-field transformers in the moderate interaction regime](https://mlsanity.com/paper/2509.25040/) (2025)
- [Softpick: No Attention Sink, No Massive Activations with Rectified Softmax](https://mlsanity.com/paper/2504.20966/) (2025)
- [Long-Context Generalization with Sparse Attention](https://mlsanity.com/paper/2506.16640/) (2025)
- [Critical attention scaling in long-context transformers](https://mlsanity.com/paper/2510.05554/) (2025)
- [TabPFN-3: Technical Report](https://mlsanity.com/paper/2605.13986/) (2026)
- [Multi-Token Attention](https://mlsanity.com/paper/2504.00927/) (2025)
- [EXAONE Tabular 1.0 : Technical Report](https://mlsanity.com/paper/2608.25774/) (2026)
- [LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligence](https://mlsanity.com/paper/2609.17488/) (2026)
- [Scale-invariant Attention](https://mlsanity.com/paper/2505.17083/) (2025)
- [Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling](https://mlsanity.com/paper/2601.12145/) (2026)
- [Homogenized Transformers](https://mlsanity.com/paper/2604.01978/) (2026)
- [Continuity and Isolation Lead to Doubts or Dilemmas in Large Language Models](https://mlsanity.com/paper/2505.10606/) (2025)
- [Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation](https://mlsanity.com/paper/2505.22842/) (2025)
- [RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling](https://mlsanity.com/paper/2507.04416/) (2025)
- [AdaSplash-2: Faster Differentiable Sparse Attention](https://mlsanity.com/paper/2604.15180/) (2026)
- [Scaling Limits of Long-Context Transformers](https://mlsanity.com/paper/2605.08505/) (2026)
- [Causilo Technical Report](https://mlsanity.com/paper/2609.22866/) (2026)
- [Sparse Attention as Compact Kernel Regression](https://mlsanity.com/paper/2601.22766/) (2026)
- [DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models](https://mlsanity.com/paper/2602.22175/) (2026)
- [SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs](https://mlsanity.com/paper/2604.07737/) (2026)
- [Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale](https://mlsanity.com/paper/2607.01538/) (2026)
- [Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrix](https://mlsanity.com/paper/2510.06685/) (2025)
- [Reduced Density Matrices Through Machine Learning](https://mlsanity.com/paper/2511.07367/) (2025)
- [TIDE: Text-Informed Dynamic Extrapolation with Step-Aware Temperature Control for Diffusion Transformers](https://mlsanity.com/paper/2603.08928/) (2026)
- [Screening Is Enough](https://mlsanity.com/paper/2604.01178/) (2026)
- [Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor](https://mlsanity.com/paper/2605.20798/) (2026)
- [ATMA: Long-Context Language Modeling via Polar Attention and Gated-Delta Compression Memory](https://mlsanity.com/paper/2606.25156/) (2026)
- [On the Mathematical Impossibility of Safe Universal Approximators](https://mlsanity.com/paper/2507.03031/) (2025)

## Abstract (arXiv)

The maximum element of the vector output by the Softmax function approaches zero as the input vector size increases. Transformer-based language models rely on Softmax to compute attention scores, causing the attention distribution to flatten as the context size grows. This reduces the model's ability to prioritize key information effectively and potentially limits its length generalization. To address this problem, we propose Scalable-Softmax (SSMax), which replaces Softmax in scenarios where the input vector size varies. SSMax can be seamlessly integrated into existing Transformer-based architectures. Experimental results in language modeling show that models using SSMax not only achieve faster loss reduction during pretraining but also significantly improve performance in long contexts and key information retrieval. Furthermore, an analysis of attention scores reveals that SSMax enables the model to focus attention on key information even in long contexts. Additionally, although models that use SSMax from the beginning of pretraining achieve better length generalization, those that have already started pretraining can still gain some of this ability by replacing Softmax in the attention layers with SSMax, either during or after pretraining.