Softmax is 1/2-Lipschitz: A tight bound across all ℓp norms
Not on the map. Why?
Abstract (arXiv)
The softmax function is a basic operator in machine learning and optimization, used in classification, attention mechanisms, reinforcement learning, game theory, and problems involving log-sum-exp terms. Existing robustness guarantees of learning models and convergence analysis of optimization algorithms typically consider the softmax operator to have a Lipschitz constant of 1 with respect to the ℓ2 norm. In this work, we prove that the softmax function is contractive with the Lipschitz constant 1/2, uniformly across all ℓp norms with p ≥ 1. We also show that the local Lipschitz constant of softmax attains 1/2 for p = 1 and p = ∞, and for p ∈ (1,∞), the constant remains strictly below 1/2 and the supremum 1/2 is achieved only in the limit. To our knowledge, this is the first comprehensive norm-uniform analysis of softmax Lipschitz continuity. We demonstrate how the sharper constant directly improves a range of existing theoretical results on robustness and convergence. We further validate the sharpness of the 1/2 Lipschitz constant of the softmax operator through empirical studies on attention-based architectures (ViT, GPT-2, Qwen3-8B) and on stochastic policies in reinforcement learning.
Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 27 Oct 2025.
Builds on
28 of 47 references.
References · 28
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2023Qwen Technical Reportcited by 3,273
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
- 2021Convex optimizationcited by 2,479
- 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
- 2021Transformers in Vision: A Surveycited by 1,009
- 2017Spectrally-normalized margin bounds for neural networkscited by 681
- 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
16 more
- 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
- 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
- 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
- 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
- 2020The Lipschitz Constant of Self-Attentioncited by 171
- 2017On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learningcited by 136
- 2022Reusing the Task-specific Classifier as a Discriminator: Discriminator-free Adversarial Domain Adaptationcited by 66
- 2021Fast Policy Extragradient Methods for Competitive Games with Entropy Regularizationcited by 61
- 2023Statistical Foundations of Prior-Data Fitted Networkscited by 59
- 2023A Survey on Transformers in Reinforcement Learningcited by 50
- 2024An Information-Theoretic Analysis of In-Context Learningcited by 39
- 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 33
- 2018On Controllable Sparse Alternatives to Softmaxcited by 26
- 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
- 2020An Analysis of the Adaptation Speed of Causal Modelscited by 9
- 2024Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real Worldcited by 1
Built on by
Citing papers per year: 2026: 18.
18 papers.
Sort:
By area:
- 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 27
- 2026Approximation Theory for Lipschitz Continuous Transformerscited by 5
- 2026Exact Attention Sensitivity and the Geometry of Transformer Stabilitycited by 3
- 2026Energy-Entropy Regularization: The True Power of Minimal Looped Transformerscited by 2
- 2026Sinkhorn doubly stochastic attention rank decay analysiscited by 2
- 2026AgentKVShift: Efficient KV Cache Reuse for Agentic Memory Systemscited by 2
- 2026Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformerscited by 1
- 2026Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updatescited by 1
- 2026Beyond LoRA: Is Sparsity-Induced Adaptation Better?cited by 1
- 2026Sharp Spectral Thresholds for Logit Fixed Pointscited by 0
- 2026Multi-Token Residual Predictioncited by 0
- 2026Smooth Partial Lotteries for Stable Randomized Selectioncited by 0
6 more
- 2026STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healingcited by 0
- 2026DnA: Denoising Attention for Visual Taskscited by 0
- 2026Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogatescited by 0
- 2026Tight Nonasymptotic Local Convergence of Sinkhorn-Knoppcited by 0
- 2026Toward a First-Principles Update Geometry for the Language-Model Headcited by 0
- 2026Read What Matters: Query-Adaptive Quantization for KV Cachescited by 0
- 2026Read What Matters: Query-Adaptive Quantization for KV Caches8 Oct 2026
- 2026Toward a First-Principles Update Geometry for the Language-Model Head23 Aug 2026
- 2026Tight Nonasymptotic Local Convergence of Sinkhorn-Knopp12 Aug 2026
- 2026Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogates31 Jul 2026
- 2026DnA: Denoising Attention for Visual Tasks25 Jun 2026
- 2026Beyond LoRA: Is Sparsity-Induced Adaptation Better?11 Jun 2026
- 2026STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing31 May 2026
- 2026Smooth Partial Lotteries for Stable Randomized Selection19 May 2026
- 2026AgentKVShift: Efficient KV Cache Reuse for Agentic Memory Systems15 May 2026
- 2026Sharp Spectral Thresholds for Logit Fixed Points15 May 2026
- 2026Multi-Token Residual Prediction12 May 2026
- 2026Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updates13 Apr 2026
6 more
- 2026A Mechanistic Analysis of Looped Reasoning Language Models13 Apr 2026
- 2026Sinkhorn doubly stochastic attention rank decay analysis9 Apr 2026
- 2026Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers11 Mar 2026
- 2026Exact Attention Sensitivity and the Geometry of Transformer Stability21 Feb 2026
- 2026Approximation Theory for Lipschitz Continuous Transformers17 Feb 2026
- 2026Energy-Entropy Regularization: The True Power of Minimal Looped Transformers14 Jan 2026
- 2026Multi-Token Residual Predictioncited by 0
- 2026Multi-Token Residual Prediction12 May 2026
- 2026Approximation Theory for Lipschitz Continuous Transformerscited by 5
- 2026Exact Attention Sensitivity and the Geometry of Transformer Stabilitycited by 3
- 2026Sinkhorn doubly stochastic attention rank decay analysiscited by 2
- 2026Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformerscited by 1
- 2026Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updatescited by 1
- 2026Beyond LoRA: Is Sparsity-Induced Adaptation Better?cited by 1
- 2026Sharp Spectral Thresholds for Logit Fixed Pointscited by 0
- 2026Smooth Partial Lotteries for Stable Randomized Selectioncited by 0
- 2026STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healingcited by 0
- 2026DnA: Denoising Attention for Visual Taskscited by 0
- 2026Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogatescited by 0
- 2026Tight Nonasymptotic Local Convergence of Sinkhorn-Knoppcited by 0
12 most cited of 14 All 14 (JSON)
- 2026Read What Matters: Query-Adaptive Quantization for KV Caches8 Oct 2026
- 2026Toward a First-Principles Update Geometry for the Language-Model Head23 Aug 2026
- 2026Tight Nonasymptotic Local Convergence of Sinkhorn-Knopp12 Aug 2026
- 2026Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogates31 Jul 2026
- 2026DnA: Denoising Attention for Visual Tasks25 Jun 2026
- 2026Beyond LoRA: Is Sparsity-Induced Adaptation Better?11 Jun 2026
- 2026STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing31 May 2026
- 2026Smooth Partial Lotteries for Stable Randomized Selection19 May 2026
- 2026Sharp Spectral Thresholds for Logit Fixed Points15 May 2026
- 2026Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updates13 Apr 2026
- 2026Sinkhorn doubly stochastic attention rank decay analysis9 Apr 2026
- 2026Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers11 Mar 2026
12 newest of 14 All 14 (JSON)