mlsanityData to 7 Oct 2026

Softmax is 1/2-Lipschitz: A tight bound across all ℓp norms

arXiv 2510.23012 ↗First posted 27 Oct 2025cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

The softmax function is a basic operator in machine learning and optimization, used in classification, attention mechanisms, reinforcement learning, game theory, and problems involving log-sum-exp terms. Existing robustness guarantees of learning models and convergence analysis of optimization algorithms typically consider the softmax operator to have a Lipschitz constant of 1 with respect to the ℓ2 norm. In this work, we prove that the softmax function is contractive with the Lipschitz constant 1/2, uniformly across all ℓp norms with p ≥ 1. We also show that the local Lipschitz constant of softmax attains 1/2 for p = 1 and p = ∞, and for p ∈ (1,∞), the constant remains strictly below 1/2 and the supremum 1/2 is achieved only in the limit. To our knowledge, this is the first comprehensive norm-uniform analysis of softmax Lipschitz continuity. We demonstrate how the sharper constant directly improves a range of existing theoretical results on robustness and convergence. We further validate the sharpness of the 1/2 Lipschitz constant of the softmax operator through empirical studies on attention-based architectures (ViT, GPT-2, Qwen3-8B) and on stochastic policies in reinforcement learning.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

28 of 47 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2021Convex optimizationcited by 2,479
  9. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  10. 2021Transformers in Vision: A Surveycited by 1,009
  11. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  12. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
16 more
  1. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  2. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  3. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  4. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  5. 2020The Lipschitz Constant of Self-Attentioncited by 171
  6. 2017On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learningcited by 136
  7. 2022Reusing the Task-specific Classifier as a Discriminator: Discriminator-free Adversarial Domain Adaptationcited by 66
  8. 2021Fast Policy Extragradient Methods for Competitive Games with Entropy Regularizationcited by 61
  9. 2023Statistical Foundations of Prior-Data Fitted Networkscited by 59
  10. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  11. 2024An Information-Theoretic Analysis of In-Context Learningcited by 39
  12. 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 33
  13. 2018On Controllable Sparse Alternatives to Softmaxcited by 26
  14. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
  15. 2020An Analysis of the Adaptation Speed of Causal Modelscited by 9
  16. 2024Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real Worldcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 18.

18 papers.

Sort:

By area: