# Theory, Analysis, and Best Practices for Sigmoid Self-Attention

arXiv: https://arxiv.org/abs/2409.04431 · HTML: https://mlsanity.com/paper/2409.04431/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2024.

## Built on by (most cited first, 30 of 64)

- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://mlsanity.com/paper/2501.01005/) (2025)
- [When Attention Sink Emerges in Language Models: An Empirical View](https://mlsanity.com/paper/2410.10781/) (2024)
- [Softmax is not Enough (for Sharp Size Generalisation)](https://mlsanity.com/paper/2410.01104/) (2024)
- [Systematic Outliers in Large Language Models](https://mlsanity.com/paper/2502.06415/) (2025)
- [CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up](https://mlsanity.com/paper/2412.16112/) (2024)
- [The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks](https://mlsanity.com/paper/2603.05498/) (2026)
- [Sliding Window Attention Training for Efficient Large Language Models](https://mlsanity.com/paper/2502.18845/) (2025)
- [A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training](https://mlsanity.com/paper/2601.22966/) (2026)
- [Attention layers provably solve single-location regression](https://mlsanity.com/paper/2410.01537/) (2024)
- [Always Skip Attention](https://mlsanity.com/paper/2505.01996/) (2025)
- [Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Models](https://mlsanity.com/paper/2409.19315/) (2024)
- [Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers](https://mlsanity.com/paper/2510.09017/) (2025)
- [RACE Attention: A Strictly Linear-Time Attention Layer for Training on Outrageously Large Contexts](https://mlsanity.com/paper/2510.04008/) (2025)
- [Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse](https://mlsanity.com/paper/2602.01203/) (2026)
- [ZeroS: Zero-Sum Linear Attention for Efficient Transformers](https://mlsanity.com/paper/2602.05230/) (2026)
- [Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective](https://mlsanity.com/paper/2502.00281/) (2025)
- [On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach](https://mlsanity.com/paper/2502.02209/) (2025)
- [PowLU: An Activation Function for Stable Pre-Training of LLMs](https://mlsanity.com/paper/2605.25704/) (2026)
- [NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMs](https://mlsanity.com/paper/2411.00151/) (2024)
- [Self-Adjust Softmax](https://mlsanity.com/paper/2502.18277/) (2025)
- [Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalization](https://mlsanity.com/paper/2506.01562/) (2025)
- [Scaling Context Requires Rethinking Attention](https://mlsanity.com/paper/2507.04239/) (2025)
- [Your Attention Matters: to Improve Model Robustness to Noise and Spurious Correlations](https://mlsanity.com/paper/2507.20453/) (2025)
- [PRISMA: Improving the Accuracy-Latency Frontier of Diffusion-based PDE Solvers Using Physics-Informed Spectral Attention](https://mlsanity.com/paper/2512.01370/) (2025)
- [Attention Needs to Focus: A Unified Perspective on Attention Allocation](https://mlsanity.com/paper/2601.00919/) (2026)
- [Legible-by-Construction: Attention and End-to-End Transformers](https://mlsanity.com/paper/2607.04319/) (2026)
- [ENSI: Efficient Non-Interactive Secure Inference for Large Language Models](https://mlsanity.com/paper/2509.09424/) (2025)
- [Sliced ReLU attention: Quasi-linear contextual expressivity via sorting](https://mlsanity.com/paper/2512.11411/) (2025)
- [Gradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutions](https://mlsanity.com/paper/2603.06248/) (2026)

## Abstract (arXiv)

Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are typically obtained as the softmax of dot products between keys and queries. Recent work has explored alternatives to softmax attention in transformers, such as ReLU and sigmoid activations. In this work, we revisit sigmoid attention and conduct an in-depth theoretical and empirical analysis. Theoretically, we prove that transformers with sigmoid attention are universal function approximators and benefit from improved regularity compared to softmax attention. Through detailed empirical analysis, we identify stabilization of large initial attention norms during the early stages of training as a crucial factor for the successful training of models with sigmoid attention, outperforming prior attempts. We also introduce FLASHSIGMOID, a hardware-aware and memory-efficient implementation of sigmoid attention yielding a 17% inference kernel speed-up over FLASHATTENTION2 on H100 GPUs. Experiments across language, vision, and speech show that properly normalized sigmoid attention matches the strong performance of softmax attention on a wide range of domains and scales, which previous attempts at sigmoid attention were unable to fully achieve. Our work unifies prior art and establishes best practices for sigmoid attention as a drop-in softmax replacement in transformers.