mlsanityData to 7 Oct 2026

Convex Bounds on the Softmax Function with Applications to Robustness Verification

arXiv 2303.01713 ↗First posted 3 Mar 2023cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

The softmax function is a ubiquitous component at the output of neural networks and increasingly in intermediate layers as well. This paper provides convex lower bounds and concave upper bounds on the softmax function, which are compatible with convex optimization formulations for characterizing neural networks and other ML models. We derive bounds using both a natural exponential-reciprocal decomposition of the softmax as well as an alternative decomposition in terms of the log-sum-exp function. The new bounds are provably and/or numerically tighter than linear bounds obtained in previous work on robustness verification of transformers. As illustrations of the utility of the bounds, we apply them to verification of transformers as well as of the robustness of predictive uncertainty estimates of deep ensembles.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 34 of its 58 references that are in the map. First posted on arXiv 3 Mar 2023.

Builds on

34 of 58 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2021Convex optimizationcited by 2,479
  5. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  6. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 739
  7. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
  8. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
  9. 2017Convolutional Sequence to Sequence Learningcited by 550
  10. 2018Efficient Neural Network Robustness Certification with General Activation Functionscited by 439
  11. 2016Safety Verification of Deep Neural Networkscited by 262
  12. 2018Towards Fast Computation of Certified Robustness for ReLU Networkscited by 254
22 more
  1. 2017Formal Verification of Piece-Wise Linear Feed-Forward Neural Networkscited by 241
  2. 2018Formal Security Analysis of Neural Networks using Symbolic Intervalscited by 185
  3. 2020Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifierscited by 183
  4. 2018Semidefinite relaxations for certifying robustness to adversarial examplescited by 178
  5. 2019Branch and Bound for Piecewise Linear Neural Network Verificationcited by 138
  6. 2020Uncertainty Quantification and Deep Ensemblescited by 138
  7. 2018A Dual Approach to Scalable Verification of Deep Networkscited by 125
  8. 2019A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networkscited by 124
  9. 2017Output Reachable Set Estimation and Verification for Multi-Layer Neural Networkscited by 79
  10. 2020Robustness Verification for Transformerscited by 76
  11. 2020Verification of Deep Convolutional Neural Networks Using ImageStarscited by 75
  12. 2018CNN-Cert: An Efficient Framework for Certifying Robustness of Convolutional Neural Networkscited by 50
  13. 2020The Convex Relaxation Barrier, Revisited: Tightened Single-Neuron Relaxations for Neural Network Verificationcited by 50
  14. 2019Optimization and Abstraction: A Synergistic Approach for Analyzing Neural Network Robustnesscited by 40
  15. 2020Reachable Polyhedral Marching (RPM): A Safety Verification Algorithm for Robotic Systems with Deep Neural Network Componentscited by 36
  16. 2018A Game-Based Approximate Verification of Deep Neural Networks with Provable Guaranteescited by 33
  17. 2022Efficient Neural Network Analysis with Sum-of-Infeasibilitiescited by 24
  18. 2020Fast Geometric Projections for Local Robustness Certificationcited by 21
  19. 2021Make Sure You're Unsure: A Framework for Verifying Probabilistic Specificationscited by 18
  20. 2016Guaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalitiescited by 16
  21. 2016One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilitiescited by 14
  22. 2022On Optimizing Back-Substitution Methods for Neural Network Verificationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 4; 2026: 8.

19 papers.

Sort:

By area:

  1. 2024Neural Network Verification with Branch-and-Bound for General Nonlinearitiescited by 47
  2. 2024Certified Robustness to Data Poisoning in Gradient-Based Trainingcited by 17
  3. 2024Certifying Robustness of Learning-Based Keypoint Detection and Pose Estimation Methodscited by 4
  4. 2025Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generationcited by 4
  5. 2023On the Robustness of Text Vectorizerscited by 2
  6. 2024Achieving the Tightest Relaxation of Sigmoids for Formal Verificationcited by 2
  7. 2026Vertex-Softmax: Tight Transformer Verification via Exact Softmax Optimizationcited by 2
  8. 2026ZonoGPT: Towards An Abstract Domain for Verifying Large GPT Modelscited by 1
  9. 2023STR-Cert: Robustness Certification for Deep Text Recognition on Deep Learning Pipelines and Vision Transformerscited by 0
  10. 2024Tightening convex relaxations of trained neural networks: a unified approach for convex and S-shaped activationscited by 0
  11. 2025Language Models That Walk the Talk: A Framework for Formal Fairness Certificatescited by 0
  12. 2025Influence-Guided Concolic Testing of Transformer Robustnesscited by 0
7 more
  1. 2025Formal Reasoning About Confidence and Automated Verification of Neural Networkscited by 0
  2. 2026The Luna Bound Propagator for Formal Analysis of Neural Networkscited by 0
  3. 2026Analyzing Symbolic Properties for DRL Agents in Systems and Networkingcited by 0
  4. 2026Towards Verified and Targeted Explanations through Formal Methodscited by 0
  5. 2026Precise Verification of Transformers through ReLU-Catalyzed Abstraction Refinementcited by 0
  6. 2026Neural Network Verification using Partial Multi-Neuron Relaxationcited by 0
  7. 2026Mining Verdict Boundaries for Neural Network Verificationcited by 0