# RobustART: Benchmarking Robustness on Architecture Design and Training Techniques

arXiv: https://arxiv.org/abs/2109.05211 · HTML: https://mlsanity.com/paper/2109.05211/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 89 of its 114 references that are in the map.
- First posted on arXiv 11 Sep 2021.

## Builds on (most cited first, 30 of 89)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)

## Built on by (most cited first, 30 of 79)

- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [Better Diffusion Models Further Improve Adversarial Training](https://mlsanity.com/paper/2302.04638/) (2023)
- [BackdoorBench: A Comprehensive Benchmark of Backdoor Learning](https://mlsanity.com/paper/2206.12654/) (2022)
- [ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond](https://mlsanity.com/paper/2202.10108/) (2022)
- [Robustness and Accuracy Could Be Reconcilable by (Proper) Definition](https://mlsanity.com/paper/2202.10103/) (2022)
- [DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planning](https://mlsanity.com/paper/2506.06659/) (2025)
- [A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking](https://mlsanity.com/paper/2302.14301/) (2023)
- [Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks](https://mlsanity.com/paper/2110.03825/) (2021)
- [SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models](https://mlsanity.com/paper/2410.18927/) (2024)
- [X-Adv: Physical Adversarial Object Attacks against X-ray Prohibited Item Detection](https://mlsanity.com/paper/2302.09491/) (2023)
- [Robust Principles: Architectural Design Principles for Adversarially Robust CNNs](https://mlsanity.com/paper/2308.16258/) (2023)
- [Are Vision Transformers Robust to Patch Perturbations?](https://mlsanity.com/paper/2111.10659/) (2021)
- [Probabilistically Robust Learning: Balancing Average- and Worst-case Performance](https://mlsanity.com/paper/2202.01136/) (2022)
- [Robust Unlearnable Examples: Protecting Data Against Adversarial Learning](https://mlsanity.com/paper/2203.14533/) (2022)
- [MambaDFuse: A Mamba-based Dual-phase Model for Multi-modality Image Fusion](https://mlsanity.com/paper/2404.08406/) (2024)
- [An Impartial Take to the CNN vs Transformer Robustness Contest](https://mlsanity.com/paper/2207.11347/) (2022)
- [GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing](https://mlsanity.com/paper/2407.00600/) (2024)
- [Towards Efficient Adversarial Training on Vision Transformers](https://mlsanity.com/paper/2207.10498/) (2022)
- [Enhancing the Self-Universality for Transferable Targeted Attacks](https://mlsanity.com/paper/2209.03716/) (2022)
- [ImageNet-Patch: A Dataset for Benchmarking Machine Learning Robustness against Adversarial Patches](https://mlsanity.com/paper/2203.04412/) (2022)
- [Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findings](https://mlsanity.com/paper/2503.15092/) (2025)
- [Exploring the Relationship between Architecture and Adversarially Robust Generalization](https://mlsanity.com/paper/2209.14105/) (2022)
- [Latent Imitator: Generating Natural Individual Discriminatory Instances for Black-Box Fairness Testing](https://mlsanity.com/paper/2305.11602/) (2023)
- [Towards Comprehensive Testing on the Robustness of Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2204.07932/) (2022)
- [Defensive Patches for Robust Recognition in the Physical World](https://mlsanity.com/paper/2204.06213/) (2022)
- [Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videos](https://mlsanity.com/paper/2301.00896/) (2023)
- [Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation](https://mlsanity.com/paper/2209.09841/) (2022)
- [RobustMQ: Benchmarking Robustness of Quantized Models](https://mlsanity.com/paper/2308.02350/) (2023)
- [BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning](https://mlsanity.com/paper/2407.19845/) (2024)
- [Towards Benchmarking and Assessing Visual Naturalness of Physical World Adversarial Attacks](https://mlsanity.com/paper/2305.12863/) (2023)

## Abstract (arXiv)

Deep neural networks (DNNs) are vulnerable to adversarial noises, which motivates the benchmark of model robustness. Existing benchmarks mainly focus on evaluating defenses, but there are no comprehensive studies of how architecture design and training techniques affect robustness. Comprehensively benchmarking their relationships is beneficial for better understanding and developing robust DNNs. Thus, we propose RobustART, the first comprehensive Robustness investigation benchmark on ImageNet regarding ARchitecture design (49 human-designed off-the-shelf architectures and 1200+ networks from neural architecture search) and Training techniques (10+ techniques, e.g., data augmentation) towards diverse noises (adversarial, natural, and system noises). Extensive experiments substantiated several insights for the first time, e.g., (1) adversarial training is effective for the robustness against all noises types for Transformers and MLP-Mixers; (2) given comparable model sizes and aligned training settings, CNNs > Transformers > MLP-Mixers on robustness against natural and system noises; Transformers > MLP-Mixers > CNNs on adversarial robustness; (3) for some light-weight architectures, increasing model sizes or using extra data cannot improve robustness. Our benchmark presents: (1) an open-source platform for comprehensive robustness evaluation; (2) a variety of pre-trained models to facilitate robustness evaluation; and (3) a new view to better understand the mechanism towards designing robust DNNs. We will continuously develop to this ecosystem for the community.