mlsanityData to 14 Oct 2026

LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustness

arXiv 2210.11620 ↗First posted 20 Oct 2022cs.LG stat.MLcited by 33

Not on the map. Why?

Abstract (arXiv)

Recent studies show that training deep neural networks (DNNs) with Lipschitz constraints are able to enhance adversarial robustness and other model properties such as stability. In this paper, we propose a layer-wise orthogonal training method (LOT) to effectively train 1-Lipschitz convolution layers via parametrizing an orthogonal matrix with an unconstrained matrix. We then efficiently compute the inverse square root of a convolution kernel by transforming the input domain to the Fourier frequency domain. On the other hand, as existing works show that semi-supervised training helps improve empirical robustness, we aim to bridge the gap and prove that semi-supervised learning also improves the certified robustness of Lipschitz-bounded models. We conduct comprehensive evaluations for LOT under different settings. We show that LOT significantly outperforms baselines regarding deterministic l2 certified robustness, and scales to deeper neural networks. Under the supervised scenario, we improve the state-of-the-art certified robustness for all architectures (e.g. from 59.04% to 63.50% on CIFAR-10 and from 32.57% to 34.59% on CIFAR-100 at radius rho = 36/255 for 40-layer networks). With semi-supervised learning over unlabelled data, we are able to improve state-of-the-art certified robustness on CIFAR-10 at rho = 108/255 from 36.04% to 42.39%. In addition, LOT consistently outperforms baselines on different model architectures with only 1/3 evaluation time.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 33 of its 36 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

33 of 36 references.

References · 33

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  4. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  5. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
  6. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  7. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 293
  8. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 293
  9. 2018Generating Adversarial Examples with Adversarial Networkscited by 258
  10. 2018Sorting out Lipschitz function approximationcited by 213
  11. 2018Spatially Transformed Adversarial Examplescited by 195
  12. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 195
21 more
  1. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 180
  2. 2018i-RevNet: Deep Invertible Networkscited by 137
  3. 2019Orthogonal Convolutional Neural Networkscited by 123
  4. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  5. 2020SoK: Certified Robustness for Deep Neural Networkscited by 105
  6. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  7. 2018The Singular Values of Convolutional Layerscited by 95
  8. 2019SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editingcited by 93
  9. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 87
  10. 2020Lipschitz constant estimation of Neural Networks via sparse polynomial optimizationcited by 87
  11. 2021Skew Orthogonal Convolutionscited by 67
  12. 2020TSS: Transformation-Specific Smoothing for Robustness Certificationcited by 49
  13. 2021Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100cited by 47
  14. 2018Characterizing Adversarial Examples Based on Spatial Consistency Information for Semantic Segmentationcited by 40
  15. 2021On the Certified Robustness for Ensemble Models and Beyondcited by 40
  16. 2021TRS: Transferability Reduced Ensemble via Encouraging Gradient Diversity and Model Smoothnesscited by 38
  17. 2020Controllable Orthogonalization in Training DNNscited by 30
  18. 2017Improved Bilinear Pooling with CNNscited by 29
  19. 2022Double Sampling Randomized Smoothingcited by 22
  20. 2018L2-Nonexpansive Neural Networkscited by 21
  21. 2020Improving Certified Robustness via Statistical Learning with Logical Reasoningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 12; 2025: 4; 2026: 4.

33 papers.

Sort:

By area:

  1. 2023A Unified Algebraic Perspective on Lipschitz Neural Networkscited by 60
  2. 2023Direct Parameterization of Lipschitz-Bounded Deep Networkscited by 51
  3. 2023Expressive Losses for Verified Robustness via Convex Combinationscited by 23
  4. 20231-Lipschitz Layers Compared: Memory, Speed, and Certifiable Robustnesscited by 16
  5. 2023Unlocking Deterministic Robustness Certification on ImageNetcited by 13
  6. 2023DP-SGD Without Clipping: The Lipschitz Neural Network Waycited by 11
  7. 2024PROSAC: Provably Safe Certification for Machine Learning Models under Adversarial Attackscited by 11
  8. 2024Group and Shuffle: Efficient Structured Orthogonal Parametrizationcited by 11
  9. 2023Robust One-Class Classification with Signed Distance Function using 1-Lipschitz Neural Networkscited by 10
  10. 2024COLEP: Certifiably Robust Learning-Reasoning Conformal Prediction via Probabilistic Circuitscited by 9
  11. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
  12. 2023Raising the Bar for Certified Adversarial Robustness with Diffusion Modelscited by 7
21 more
  1. 2024Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activationscited by 7
  2. 2024Spectrum Extraction and Clipping for Implicitly Linear Layerscited by 7
  3. 2025Enhancing Certified Robustness via Block Reflector Orthogonal Layers and Logit Annealing Losscited by 5
  4. 2023A Recipe for Improved Certifiable Robustnesscited by 4
  5. 2024SPLITZ: Certifiable Robustness via Split Lipschitz Randomized Smoothingcited by 4
  6. 2023Certified Robust Models with Slack Control and Large Lipschitz Constantscited by 3
  7. 2024On Using Certified Training towards Empirical Robustnesscited by 3
  8. 2024Intriguing Properties of Robust Classificationcited by 3
  9. 2025LipShiFT: A Certifiably Robust Shift-based Vision Transformercited by 3
  10. 2023Towards Better Certified Segmentation via Diffusion Modelscited by 2
  11. 2024COMMIT: Certifying Robustness of Multi-Sensor Fusion Systems against Semantic Attackscited by 2
  12. 2026LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Modelscited by 2
  13. 2023Towards Large Certified Radius in Randomized Smoothing using Quasiconcave Optimizationcited by 1
  14. 2023Post-train Black-box Defense via Bayesian Boundary Correctioncited by 1
  15. 2024Towards Generalized Certified Robustness with Multi-Norm Trainingcited by 1
  16. 2024LOTOS: Layer-wise Orthogonalization for Training Robust Ensemblescited by 1
  17. 2026Certified vs. Empirical Adversarial Robust-ness via Hybrid Convolutions with Attention Stochasticitycited by 1
  18. 2024Inference Privacy: Properties and Mechanismscited by 0
  19. 2025Almost Right: Making First-Layer Kernels Nearly Orthogonal Improves Model Generalizationcited by 0
  20. 2026LiST: Lipschitz Scaling Training for Robust and Calibrated Neural Networkscited by 0
  21. 2026Adversarial Detection for Pre-trained Classifiers via Lipschitz-Margin Analysis with Certificationcited by 0