mlsanityData to 7 Oct 2026

Improving Robustness by Enhancing Weak Subnets

arXiv 2201.12765 ↗First posted 30 Jan 2022cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Despite their success, deep networks have been shown to be highly susceptible to perturbations, often causing significant drops in accuracy. In this paper, we investigate model robustness on perturbed inputs by studying the performance of internal sub-networks (subnets). Interestingly, we observe that most subnets show particularly poor robustness against perturbations. More importantly, these weak subnets are correlated with the overall lack of robustness. Tackling this phenomenon, we propose a new training procedure that identifies and enhances weak subnets (EWS) to improve robustness. Specifically, we develop a search algorithm to find particularly weak subnets and explicitly strengthen them via knowledge distillation from the full network. We show that EWS greatly improves both robustness against corrupted images as well as accuracy on clean data. Being complementary to popular data augmentation methods, EWS consistently improves robustness when combined with these approaches. To highlight the flexibility of our approach, we combine EWS also with popular adversarial training methods resulting in improved adversarial robustness.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 78 of its 93 references that are in the map. First posted on arXiv 30 Jan 2022.

Builds on

78 of 93 references.

References · 78

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  9. 2016Wide Residual Networkscited by 3,057
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  12. 2016Identity Mappings in Deep Residual Networkscited by 1,960
38 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  3. 2020Knowledge Distillation: A Surveycited by 1,578
  4. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  5. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  6. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  7. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  8. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  9. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  10. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  11. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  12. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  13. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  14. 2020Overfitting in adversarially robust deep learningcited by 504
  15. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  16. 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 459
  17. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  18. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  19. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 411
  20. 2014Deeply-Supervised Netscited by 364
  21. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 350
  22. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
  23. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 275
  24. 2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 242
  25. 2020Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shiftcited by 235
  26. 2019Adversarial Attacks and Defenses in Images, Graphs and Text: A Reviewcited by 234
  27. 2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 201
  28. 2020Bag of Tricks for Adversarial Trainingcited by 199
  29. 2018MMA Training: Direct Input Space Margin Maximization through Adversarial Trainingcited by 171
  30. 2019Are Labels Required for Improving Adversarial Robustness?cited by 162
  31. 2021Towards Robust Vision Transformercited by 137
  32. 2018Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attackcited by 123
  33. 2020Boosting Adversarial Training with Hypersphere Embeddingcited by 97
  34. 2019Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Trainingcited by 95
  35. 2020Learnable Boundary Guided Adversarial Trainingcited by 93
  36. 2018Curriculum Adversarial Trainingcited by 89
  37. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 77
  38. 2020Opportunities and Challenges in Deep Learning Adversarial Robustness: A Surveycited by 71

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 3; 2024: 2; 2025: 0; 2026: 2.

12 papers.

  1. 2023Robustifying Token Attention for Vision Transformerscited by 19
  2. 2022Towards Lightweight Super-Resolution with Dual Regression Learningcited by 13
  3. 2023HybridAugment++: Unified Frequency Spectra Perturbations for Model Robustnesscited by 12
  4. 2022Improving Fine-tuning of Self-supervised Models with Contrastive Initializationcited by 10
  5. 2022Downscaled Representation Matters: Improving Image Rescaling with Collaborative Downscaled Imagescited by 8
  6. 2022Boosting Semi-Supervised Learning with Contrastive Complementary Labelingcited by 5
  7. 2024CycleCrash: A Dataset of Bicycle Collision Videos for Collision Prediction and Analysiscited by 3
  8. 2024RobustFormer: Noise-Robust Pre-training for images and videoscited by 2
  9. 2022Text-Aware Dual Routing Network for Visual Question Answeringcited by 0
  10. 2023A Theoretical Perspective on Subnetwork Contributions to Adversarial Robustnesscited by 0
  11. 2026Robustness Emerges Early in Training Dynamics, but Is Not Preservedcited by 0
  12. 2026Suppress and Diversify: Refining Robust Pathways for Corruption Robustnesscited by 0