# Robust Principles: Architectural Design Principles for Adversarially Robust CNNs

arXiv: https://arxiv.org/abs/2308.16258 · HTML: https://mlsanity.com/paper/2308.16258/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 57 references that are in the map.
- First posted on arXiv 30 Aug 2023.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Fast is better than free: Revisiting adversarial training](https://mlsanity.com/paper/2001.03994/) (2020)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Diffusion Models for Adversarial Purification](https://mlsanity.com/paper/2205.07460/) (2022)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [Better Diffusion Models Further Improve Adversarial Training](https://mlsanity.com/paper/2302.04638/) (2023)
- [Are Transformers More Robust Than CNNs?](https://mlsanity.com/paper/2111.05464/) (2021)
- [MMA Training: Direct Input Space Margin Maximization through Adversarial Training](https://mlsanity.com/paper/1812.02637/) (2018)
- [On the Convergence and Robustness of Adversarial Training](https://mlsanity.com/paper/2112.08304/) (2021)

## Built on by (most cited first, 30 of 46)

- [Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack](https://mlsanity.com/paper/2402.01109/) (2024)
- [Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey](https://mlsanity.com/paper/2409.18169/) (2024)
- [Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies](https://mlsanity.com/paper/2404.09349/) (2024)
- [Securely Fine-tuning Pre-trained Encoders Against Adversarial Examples](https://mlsanity.com/paper/2403.10801/) (2024)
- [MeanSparse: Post-Training Robustness Enhancement Through Mean-Centered Feature Sparsification](https://mlsanity.com/paper/2406.05927/) (2024)
- [MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers](https://mlsanity.com/paper/2402.02263/) (2024)
- [One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIP](https://mlsanity.com/paper/2505.19840/) (2025)
- [Defenses in Adversarial Machine Learning: A Survey](https://mlsanity.com/paper/2312.08890/) (2023)
- [Large Reasoning Models Learn Better Alignment from Flawed Thinking](https://mlsanity.com/paper/2510.00938/) (2025)
- [MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness](https://mlsanity.com/paper/2312.04960/) (2023)
- [Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness](https://mlsanity.com/paper/2408.05446/) (2024)
- [Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler](https://mlsanity.com/paper/2510.27172/) (2025)
- [Shedding More Light on Robust Classifiers under the lens of Energy-based Models](https://mlsanity.com/paper/2407.06315/) (2024)
- [Adversarial Guided Diffusion Models for Adversarial Purification](https://mlsanity.com/paper/2403.16067/) (2024)
- [σ-zero: Gradient-based Optimization of ℓ0-norm Adversarial Examples](https://mlsanity.com/paper/2402.01879/) (2024)
- [Assessing Robustness via Score-Based Adversarial Image Generation](https://mlsanity.com/paper/2310.04285/) (2023)
- [High-Performance Transformers for Table Structure Recognition Need Early Convolutions](https://mlsanity.com/paper/2311.05565/) (2023)
- [A Random Ensemble of Encrypted Vision Transformers for Adversarially Robust Defense](https://mlsanity.com/paper/2402.07183/) (2024)
- [A Notion of Uniqueness for the Adversarial Bayes Classifier](https://mlsanity.com/paper/2404.16956/) (2024)
- [Data-Driven Lipschitz Continuity: A Cost-Effective Approach to Improve Adversarial Robustness](https://mlsanity.com/paper/2406.19622/) (2024)
- [Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks](https://mlsanity.com/paper/2411.15210/) (2024)
- [GreedyPixel: Fine-Grained Black-Box Adversarial Attack Via Greedy Algorithm](https://mlsanity.com/paper/2501.14230/) (2025)
- [Large-Scale Dataset Pruning in Adversarial Training through Data Importance Extrapolation](https://mlsanity.com/paper/2406.13283/) (2024)
- [Criticality Leveraged Adversarial Training (CLAT) for Boosted Performance via Parameter Efficiency](https://mlsanity.com/paper/2408.10204/) (2024)
- [On the Robustness of Adversarial Training Against Uncertainty Attacks](https://mlsanity.com/paper/2410.21952/) (2024)
- [RobustBlack: Challenging Black-Box Adversarial Attacks on State-of-the-Art Defenses](https://mlsanity.com/paper/2412.20987/) (2024)
- [Understanding Adversarial Training with Energy-based Models](https://mlsanity.com/paper/2505.22486/) (2025)
- [IRAD: Implicit Representation-driven Image Resampling against Adversarial Attacks](https://mlsanity.com/paper/2310.11890/) (2023)
- [Adversarial Examples are Misaligned in Diffusion Model Manifolds](https://mlsanity.com/paper/2401.06637/) (2024)
- [RAMP: Boosting Adversarial Robustness Against Multiple lp Perturbations for Universal Robustness](https://mlsanity.com/paper/2402.06827/) (2024)

## Abstract (arXiv)

Our research aims to unify existing works' diverging opinions on how architectural components affect the adversarial robustness of CNNs. To accomplish our goal, we synthesize a suite of three generalizable robust architectural design principles: (a) optimal range for depth and width configurations, (b) preferring convolutional over patchify stem stage, and (c) robust residual block design through adopting squeeze and excitation blocks and non-parametric smooth activation functions. Through extensive experiments across a wide spectrum of dataset scales, adversarial training methods, model parameters, and network design spaces, our principles consistently and markedly improve AutoAttack accuracy: 1-3 percentage points (pp) on CIFAR-10 and CIFAR-100, and 4-9 pp on ImageNet. The code is publicly available at https://github.com/poloclub/robust-principles.