# LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustness

arXiv: https://arxiv.org/abs/2210.11620 · HTML: https://mlsanity.com/paper/2210.11620/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 36 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 30 of 33)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Unlabeled Data Improves Adversarial Robustness](https://mlsanity.com/paper/1905.13736/) (2019)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks](https://mlsanity.com/paper/1906.04893/) (2019)
- [Generating Adversarial Examples with Adversarial Networks](https://mlsanity.com/paper/1801.02610/) (2018)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Spatially Transformed Adversarial Examples](https://mlsanity.com/paper/1801.02612/) (2018)
- [Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks](https://mlsanity.com/paper/1802.04034/) (2018)
- [Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data](https://mlsanity.com/paper/2010.03622/) (2020)
- [i-RevNet: Deep Invertible Networks](https://mlsanity.com/paper/1802.07088/) (2018)
- [Orthogonal Convolutional Neural Networks](https://mlsanity.com/paper/1911.12207/) (2019)
- [Feature Purification: How Adversarial Training Performs Robust Deep Learning](https://mlsanity.com/paper/2005.10190/) (2020)
- [SoK: Certified Robustness for Deep Neural Networks](https://mlsanity.com/paper/2009.04131/) (2020)
- [Orthogonalizing Convolutional Layers with the Cayley Transform](https://mlsanity.com/paper/2104.07167/) (2021)
- [The Singular Values of Convolutional Layers](https://mlsanity.com/paper/1805.10408/) (2018)
- [SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing](https://mlsanity.com/paper/1906.07927/) (2019)
- [Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks](https://mlsanity.com/paper/1911.00937/) (2019)
- [Lipschitz constant estimation of Neural Networks via sparse polynomial optimization](https://mlsanity.com/paper/2004.08688/) (2020)
- [Skew Orthogonal Convolutions](https://mlsanity.com/paper/2105.11417/) (2021)
- [TSS: Transformation-Specific Smoothing for Robustness Certification](https://mlsanity.com/paper/2002.12398/) (2020)
- [Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100](https://mlsanity.com/paper/2108.04062/) (2021)
- [Characterizing Adversarial Examples Based on Spatial Consistency Information for Semantic Segmentation](https://mlsanity.com/paper/1810.05162/) (2018)
- [On the Certified Robustness for Ensemble Models and Beyond](https://mlsanity.com/paper/2107.10873/) (2021)
- [TRS: Transferability Reduced Ensemble via Encouraging Gradient Diversity and Model Smoothness](https://mlsanity.com/paper/2104.00671/) (2021)
- [Controllable Orthogonalization in Training DNNs](https://mlsanity.com/paper/2004.00917/) (2020)
- [Improved Bilinear Pooling with CNNs](https://mlsanity.com/paper/1707.06772/) (2017)

## Built on by (most cited first, 30 of 33)

- [A Unified Algebraic Perspective on Lipschitz Neural Networks](https://mlsanity.com/paper/2303.03169/) (2023)
- [Direct Parameterization of Lipschitz-Bounded Deep Networks](https://mlsanity.com/paper/2301.11526/) (2023)
- [Expressive Losses for Verified Robustness via Convex Combinations](https://mlsanity.com/paper/2305.13991/) (2023)
- [1-Lipschitz Layers Compared: Memory, Speed, and Certifiable Robustness](https://mlsanity.com/paper/2311.16833/) (2023)
- [Unlocking Deterministic Robustness Certification on ImageNet](https://mlsanity.com/paper/2301.12549/) (2023)
- [DP-SGD Without Clipping: The Lipschitz Neural Network Way](https://mlsanity.com/paper/2305.16202/) (2023)
- [PROSAC: Provably Safe Certification for Machine Learning Models under Adversarial Attacks](https://mlsanity.com/paper/2402.02629/) (2024)
- [Group and Shuffle: Efficient Structured Orthogonal Parametrization](https://mlsanity.com/paper/2406.10019/) (2024)
- [Robust One-Class Classification with Signed Distance Function using 1-Lipschitz Neural Networks](https://mlsanity.com/paper/2303.01978/) (2023)
- [COLEP: Certifiably Robust Learning-Reasoning Conformal Prediction via Probabilistic Circuits](https://mlsanity.com/paper/2403.11348/) (2024)
- [An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures](https://mlsanity.com/paper/2501.07930/) (2025)
- [Raising the Bar for Certified Adversarial Robustness with Diffusion Models](https://mlsanity.com/paper/2305.10388/) (2023)
- [Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations](https://mlsanity.com/paper/2401.14033/) (2024)
- [Spectrum Extraction and Clipping for Implicitly Linear Layers](https://mlsanity.com/paper/2402.16017/) (2024)
- [Enhancing Certified Robustness via Block Reflector Orthogonal Layers and Logit Annealing Loss](https://mlsanity.com/paper/2505.15174/) (2025)
- [A Recipe for Improved Certifiable Robustness](https://mlsanity.com/paper/2310.02513/) (2023)
- [SPLITZ: Certifiable Robustness via Split Lipschitz Randomized Smoothing](https://mlsanity.com/paper/2407.02811/) (2024)
- [Certified Robust Models with Slack Control and Large Lipschitz Constants](https://mlsanity.com/paper/2309.06166/) (2023)
- [On Using Certified Training towards Empirical Robustness](https://mlsanity.com/paper/2410.01617/) (2024)
- [Intriguing Properties of Robust Classification](https://mlsanity.com/paper/2412.04245/) (2024)
- [LipShiFT: A Certifiably Robust Shift-based Vision Transformer](https://mlsanity.com/paper/2503.14751/) (2025)
- [Towards Better Certified Segmentation via Diffusion Models](https://mlsanity.com/paper/2306.09949/) (2023)
- [COMMIT: Certifying Robustness of Multi-Sensor Fusion Systems against Semantic Attacks](https://mlsanity.com/paper/2403.02329/) (2024)
- [LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models](https://mlsanity.com/paper/2601.18513/) (2026)
- [Towards Large Certified Radius in Randomized Smoothing using Quasiconcave Optimization](https://mlsanity.com/paper/2302.00209/) (2023)
- [Post-train Black-box Defense via Bayesian Boundary Correction](https://mlsanity.com/paper/2306.16979/) (2023)
- [Towards Generalized Certified Robustness with Multi-Norm Training](https://mlsanity.com/paper/2410.03000/) (2024)
- [LOTOS: Layer-wise Orthogonalization for Training Robust Ensembles](https://mlsanity.com/paper/2410.05136/) (2024)
- [Certified vs. Empirical Adversarial Robust-ness via Hybrid Convolutions with Attention Stochasticity](https://mlsanity.com/paper/2605.01519/) (2026)
- [Inference Privacy: Properties and Mechanisms](https://mlsanity.com/paper/2411.18746/) (2024)

## Abstract (arXiv)

Recent studies show that training deep neural networks (DNNs) with Lipschitz constraints are able to enhance adversarial robustness and other model properties such as stability. In this paper, we propose a layer-wise orthogonal training method (LOT) to effectively train 1-Lipschitz convolution layers via parametrizing an orthogonal matrix with an unconstrained matrix. We then efficiently compute the inverse square root of a convolution kernel by transforming the input domain to the Fourier frequency domain. On the other hand, as existing works show that semi-supervised training helps improve empirical robustness, we aim to bridge the gap and prove that semi-supervised learning also improves the certified robustness of Lipschitz-bounded models. We conduct comprehensive evaluations for LOT under different settings. We show that LOT significantly outperforms baselines regarding deterministic l2 certified robustness, and scales to deeper neural networks. Under the supervised scenario, we improve the state-of-the-art certified robustness for all architectures (e.g. from 59.04% to 63.50% on CIFAR-10 and from 32.57% to 34.59% on CIFAR-100 at radius rho = 36/255 for 40-layer networks). With semi-supervised learning over unlabelled data, we are able to improve state-of-the-art certified robustness on CIFAR-10 at rho = 108/255 from 36.04% to 42.39%. In addition, LOT consistently outperforms baselines on different model architectures with only 1/3 evaluation time.