mlsanityData to 7 Oct 2026

LipShiFT: A Certifiably Robust Shift-based Vision Transformer

arXiv 2503.14751 ↗First posted 18 Mar 2025cs.LG cs.AI cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Deriving tight Lipschitz bounds for transformer-based architectures presents a significant challenge. The large input sizes and high-dimensional attention modules typically prove to be crucial bottlenecks during the training process and leads to sub-optimal results. Our research highlights practical constraints of these methods in vision tasks. We find that Lipschitz-based margin training acts as a strong regularizer while restricting weights in successive layers of the model. Focusing on a Lipschitz continuous variant of the ShiftViT model, we address significant training challenges for transformer-based architectures under norm-constrained input setting. We provide an upper bound estimate for the Lipschitz constants of this model using the l2 norm on common image classification datasets. Ultimately, we demonstrate that our method scales to larger models and advances the state-of-the-art in certified robustness for transformer-based architectures.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

37 of 51 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  11. 2018Group Normalizationcited by 1,514
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
25 more
  1. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  2. 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 459
  3. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  4. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  5. 2018Sorting out Lipschitz function approximationcited by 211
  6. 2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 201
  7. 2020The Lipschitz Constant of Self-Attentioncited by 171
  8. 2021Globally-Robust Neural Networkscited by 106
  9. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  10. 2021On the Adversarial Robustness of Vision Transformerscited by 104
  11. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  12. 2018Generalizable Adversarial Training via Spectral Normalizationcited by 68
  13. 2021Skew Orthogonal Convolutionscited by 67
  14. 2021Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Boundscited by 64
  15. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 59
  16. 2023A Unified Algebraic Perspective on Lipschitz Neural Networkscited by 58
  17. 2023Direct Parameterization of Lipschitz-Bounded Deep Networkscited by 49
  18. 2021Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100cited by 47
  19. 2021A Review of Formal Methods applied to Machine Learningcited by 33
  20. 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 33
  21. 2022LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustnesscited by 32
  22. 2022When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanismcited by 30
  23. 20231-Lipschitz Layers Compared: Memory, Speed, and Certifiable Robustnesscited by 15
  24. 2023Lipschitz-bounded 1D convolutional neural networks using the Cayley transform and the controllability Gramiancited by 9
  25. 2023CertViT: Certified Robustness of Pre-Trained Vision Transformerscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
  2. 2026Approximation Theory for Lipschitz Continuous Transformerscited by 5
  3. 2025Understanding Sensitivity of Differential Attention through the Lens of Adversarial Robustnesscited by 0