# Improving Robustness by Enhancing Weak Subnets

arXiv: https://arxiv.org/abs/2201.12765 · HTML: https://mlsanity.com/paper/2201.12765/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 78 of its 93 references that are in the map.
- First posted on arXiv 30 Jan 2022.

## Builds on (most cited first, 30 of 78)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [Overfitting in adversarially robust deep learning](https://mlsanity.com/paper/2002.11569/) (2020)
- [Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/1801.00553/) (2018)
- [Improving robustness against common corruptions by covariate shift adaptation](https://mlsanity.com/paper/2006.16971/) (2020)
- [Unlabeled Data Improves Adversarial Robustness](https://mlsanity.com/paper/1905.13736/) (2019)

## Built on by (most cited first, 13 of 13)

- [Robustifying Token Attention for Vision Transformers](https://mlsanity.com/paper/2303.11126/) (2023)
- [Towards Lightweight Super-Resolution with Dual Regression Learning](https://mlsanity.com/paper/2207.07929/) (2022)
- [HybridAugment++: Unified Frequency Spectra Perturbations for Model Robustness](https://mlsanity.com/paper/2307.11823/) (2023)
- [Improving Fine-tuning of Self-supervised Models with Contrastive Initialization](https://mlsanity.com/paper/2208.00238/) (2022)
- [Downscaled Representation Matters: Improving Image Rescaling with Collaborative Downscaled Images](https://mlsanity.com/paper/2211.10643/) (2022)
- [Boosting Semi-Supervised Learning with Contrastive Complementary Labeling](https://mlsanity.com/paper/2212.06643/) (2022)
- [CycleCrash: A Dataset of Bicycle Collision Videos for Collision Prediction and Analysis](https://mlsanity.com/paper/2409.19942/) (2024)
- [RobustFormer: Noise-Robust Pre-training for images and videos](https://mlsanity.com/paper/2411.13040/) (2024)
- [Text-Aware Dual Routing Network for Visual Question Answering](https://mlsanity.com/paper/2211.14450/) (2022)
- [A Theoretical Perspective on Subnetwork Contributions to Adversarial Robustness](https://mlsanity.com/paper/2307.03803/) (2023)
- [Provable Repair of Deep Neural Network Defects by Preimage Synthesis and Property Refinement](https://mlsanity.com/paper/2511.07741/) (2025)
- [Robustness Emerges Early in Training Dynamics, but Is Not Preserved](https://mlsanity.com/paper/2608.04442/) (2026)
- [Suppress and Diversify: Refining Robust Pathways for Corruption Robustness](https://mlsanity.com/paper/2608.06712/) (2026)

## Abstract (arXiv)

Despite their success, deep networks have been shown to be highly susceptible to perturbations, often causing significant drops in accuracy. In this paper, we investigate model robustness on perturbed inputs by studying the performance of internal sub-networks (subnets). Interestingly, we observe that most subnets show particularly poor robustness against perturbations. More importantly, these weak subnets are correlated with the overall lack of robustness. Tackling this phenomenon, we propose a new training procedure that identifies and enhances weak subnets (EWS) to improve robustness. Specifically, we develop a search algorithm to find particularly weak subnets and explicitly strengthen them via knowledge distillation from the full network. We show that EWS greatly improves both robustness against corrupted images as well as accuracy on clean data. Being complementary to popular data augmentation methods, EWS consistently improves robustness when combined with these approaches. To highlight the flexibility of our approach, we combine EWS also with popular adversarial training methods resulting in improved adversarial robustness.