# Generalist: Decoupling Natural and Robust Generalization

arXiv: https://arxiv.org/abs/2303.13813 · HTML: https://mlsanity.com/paper/2303.13813/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 46 references that are in the map.
- First posted on arXiv 24 Mar 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks](https://mlsanity.com/paper/1704.01155/) (2017)
- [Unlabeled Data Improves Adversarial Robustness](https://mlsanity.com/paper/1905.13736/) (2019)
- [Attacks Which Do Not Kill Training Make Adversarial Learning Stronger](https://mlsanity.com/paper/2002.11242/) (2020)
- [Fixing Data Augmentation to Improve Adversarial Robustness](https://mlsanity.com/paper/2103.01946/) (2021)
- [Are Labels Required for Improving Adversarial Robustness?](https://mlsanity.com/paper/1905.13725/) (2019)
- [Understanding and Mitigating the Tradeoff Between Robustness and Accuracy](https://mlsanity.com/paper/2002.10716/) (2020)
- [On the Convergence and Robustness of Adversarial Training](https://mlsanity.com/paper/2112.08304/) (2021)
- [Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification](https://mlsanity.com/paper/1611.05377/) (2016)
- [Adversarial Vertex Mixup: Toward Better Adversarially Robust Generalization](https://mlsanity.com/paper/2003.02484/) (2020)
- [Deep Multi-task Representation Learning: A Tensor Factorisation Approach](https://mlsanity.com/paper/1605.06391/) (2016)
- [When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture](https://mlsanity.com/paper/2210.07540/) (2022)
- [Robustness to Adversarial Perturbations in Learning from Incomplete Data](https://mlsanity.com/paper/1905.13021/) (2019)
- [Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy](https://mlsanity.com/paper/1906.06784/) (2019)
- [Transferable, Controllable, and Inconspicuous Adversarial Attacks on Person Re-identification With Deep Mis-Ranking](https://mlsanity.com/paper/2004.04199/) (2020)

## Built on by (most cited first, 14 of 14)

- [Fight Back Against Jailbreaking via Prompt Adversarial Tuning](https://mlsanity.com/paper/2402.06255/) (2024)
- [CFA: Class-wise Calibrated Fair Adversarial Training](https://mlsanity.com/paper/2303.14460/) (2023)
- [Balance, Imbalance, and Rebalance: Understanding Robust Overfitting from a Minimax Game Perspective](https://mlsanity.com/paper/2310.19360/) (2023)
- [Sharpness-Aware Minimization Alone can Improve Adversarial Robustness](https://mlsanity.com/paper/2305.05392/) (2023)
- [MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness](https://mlsanity.com/paper/2312.04960/) (2023)
- [Robustly Optimized Deep Feature Decoupling Network for Fatty Liver Diseases Detection](https://mlsanity.com/paper/2406.17338/) (2024)
- [HoneyTrap: Deceiving Large Language Model Attackers to Honeypot Traps with Resilient Multi-Agent Defense](https://mlsanity.com/paper/2601.04034/) (2026)
- [Splitting the Difference on Adversarial Training](https://mlsanity.com/paper/2310.02480/) (2023)
- [FAIR-TAT: Improving Model Fairness Using Targeted Adversarial Training](https://mlsanity.com/paper/2410.23142/) (2024)
- [Generalist++: A Meta-learning Framework for Mitigating Trade-off in Adversarial Training](https://mlsanity.com/paper/2510.13361/) (2025)
- [Parameter Interpolation Adversarial Training for Robust Image Classification](https://mlsanity.com/paper/2511.00836/) (2025)
- [Revisiting the Relationship between Adversarial and Clean Training: Why Clean Training Can Make Adversarial Training Better](https://mlsanity.com/paper/2504.00038/) (2025)
- [Proxy Robustness in Vision Language Models is Effortlessly Transferable](https://mlsanity.com/paper/2601.12865/) (2026)
- [Unifying Adversarially Robust Model Experts in Vision-Language Models](https://mlsanity.com/paper/2607.27897/) (2026)

## Abstract (arXiv)

Deep neural networks obtained by standard training have been constantly plagued by adversarial examples. Although adversarial training demonstrates its capability to defend against adversarial examples, unfortunately, it leads to an inevitable drop in the natural generalization. To address the issue, we decouple the natural generalization and the robust generalization from joint training and formulate different training strategies for each one. Specifically, instead of minimizing a global loss on the expectation over these two generalization errors, we propose a bi-expert framework called Generalist where we simultaneously train base learners with task-aware strategies so that they can specialize in their own fields. The parameters of base learners are collected and combined to form a global learner at intervals during the training process. The global learner is then distributed to the base learners as initialized parameters for continued training. Theoretically, we prove that the risks of Generalist will get lower once the base learners are well trained. Extensive experiments verify the applicability of Generalist to achieve high accuracy on natural examples while maintaining considerable robustness to adversarial ones. Code is available at https://github.com/PKU-ML/Generalist.