# Softmax is 1/2-Lipschitz: A tight bound across all ℓp norms

arXiv: https://arxiv.org/abs/2510.23012 · HTML: https://mlsanity.com/paper/2510.23012/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 27 Oct 2025.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks](https://mlsanity.com/paper/1906.04893/) (2019)
- [Bridging the Gap Between Value and Policy Based Reinforcement Learning](https://mlsanity.com/paper/1702.08892/) (2017)
- [The Lipschitz Constant of Self-Attention](https://mlsanity.com/paper/2006.04710/) (2020)
- [On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning](https://mlsanity.com/paper/1704.00805/) (2017)
- [Reusing the Task-specific Classifier as a Discriminator: Discriminator-free Adversarial Domain Adaptation](https://mlsanity.com/paper/2204.03838/) (2022)
- [Statistical Foundations of Prior-Data Fitted Networks](https://mlsanity.com/paper/2305.11097/) (2023)
- [Fast Policy Extragradient Methods for Competitive Games with Entropy Regularization](https://mlsanity.com/paper/2105.15186/) (2021)
- [A Survey on Transformers in Reinforcement Learning](https://mlsanity.com/paper/2301.03044/) (2023)
- [An Information-Theoretic Analysis of In-Context Learning](https://mlsanity.com/paper/2401.15530/) (2024)
- [LipsFormer: Introducing Lipschitz Continuity to Vision Transformers](https://mlsanity.com/paper/2304.09856/) (2023)
- [On Controllable Sparse Alternatives to Softmax](https://mlsanity.com/paper/1810.11975/) (2018)
- [Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers](https://mlsanity.com/paper/2507.07814/) (2025)
- [An Analysis of the Adaptation Speed of Causal Models](https://mlsanity.com/paper/2005.09136/) (2020)
- [Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real World](https://mlsanity.com/paper/2403.20047/) (2024)

## Built on by (most cited first, 22 of 22)

- [A Mechanistic Analysis of Looped Reasoning Language Models](https://mlsanity.com/paper/2604.11791/) (2026)
- [Approximation Theory for Lipschitz Continuous Transformers](https://mlsanity.com/paper/2602.15503/) (2026)
- [Exact Attention Sensitivity and the Geometry of Transformer Stability](https://mlsanity.com/paper/2602.18849/) (2026)
- [Energy-Entropy Regularization: The True Power of Minimal Looped Transformers](https://mlsanity.com/paper/2601.09588/) (2026)
- [Sinkhorn doubly stochastic attention rank decay analysis](https://mlsanity.com/paper/2604.07925/) (2026)
- [AgentKVShift: Efficient KV Cache Reuse for Agentic Memory Systems](https://mlsanity.com/paper/2607.21604/) (2026)
- [Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers](https://mlsanity.com/paper/2603.11161/) (2026)
- [Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updates](https://mlsanity.com/paper/2604.13130/) (2026)
- [Beyond LoRA: Is Sparsity-Induced Adaptation Better?](https://mlsanity.com/paper/2606.13767/) (2026)
- [Sharp Spectral Thresholds for Logit Fixed Points](https://mlsanity.com/paper/2605.15651/) (2026)
- [Multi-Token Residual Prediction](https://mlsanity.com/paper/2605.18817/) (2026)
- [Smooth Partial Lotteries for Stable Randomized Selection](https://mlsanity.com/paper/2605.20069/) (2026)
- [STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing](https://mlsanity.com/paper/2606.01126/) (2026)
- [DnA: Denoising Attention for Visual Tasks](https://mlsanity.com/paper/2606.27372/) (2026)
- [Lipschitz Continuity in Deep Learning: A Systematic Review of Theoretical Foundations, Estimation Methods, Regularization Approaches, and Certifiable Robustness](https://mlsanity.com/paper/2607.16329/) (2026)
- [Weak-to-Strong Learning in Decision Making](https://mlsanity.com/paper/2607.18467/) (2026)
- [Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogates](https://mlsanity.com/paper/2607.28959/) (2026)
- [Tight Nonasymptotic Local Convergence of Sinkhorn-Knopp](https://mlsanity.com/paper/2608.11760/) (2026)
- [Toward a First-Principles Update Geometry for the Language-Model Head](https://mlsanity.com/paper/2608.22253/) (2026)
- [MuonIO: Principled Norm-Aware Descent for Embedding Tables and Language Model Heads](https://mlsanity.com/paper/2610.02705/) (2026)
- [On the Trade-off Between Information Loss and Generalization in Sparse Attention](https://mlsanity.com/paper/2610.04424/) (2026)
- [Read What Matters: Query-Adaptive Quantization for KV Caches](https://mlsanity.com/paper/2610.11245/) (2026)

## Abstract (arXiv)

The softmax function is a basic operator in machine learning and optimization, used in classification, attention mechanisms, reinforcement learning, game theory, and problems involving log-sum-exp terms. Existing robustness guarantees of learning models and convergence analysis of optimization algorithms typically consider the softmax operator to have a Lipschitz constant of 1 with respect to the ℓ2 norm. In this work, we prove that the softmax function is contractive with the Lipschitz constant 1/2, uniformly across all ℓp norms with p ≥ 1. We also show that the local Lipschitz constant of softmax attains 1/2 for p = 1 and p = ∞, and for p ∈ (1,∞), the constant remains strictly below 1/2 and the supremum 1/2 is achieved only in the limit. To our knowledge, this is the first comprehensive norm-uniform analysis of softmax Lipschitz continuity. We demonstrate how the sharper constant directly improves a range of existing theoretical results on robustness and convergence. We further validate the sharpness of the 1/2 Lipschitz constant of the softmax operator through empirical studies on attention-based architectures (ViT, GPT-2, Qwen3-8B) and on stochastic policies in reinforcement learning.