mlsanityData to 14 Oct 2026

Improved OOD Generalization via Adversarial Training and Pre-training

arXiv 2105.11144 ↗First posted 24 May 2021cs.LGcited by 66

Not on the map. Why?

Abstract (arXiv)

Recently, learning a model that generalizes well on out-of-distribution (OOD) data has attracted great attention in the machine learning community. In this paper, after defining OOD generalization via Wasserstein distance, we theoretically show that a model robust to input perturbation generalizes well on OOD data. Inspired by previous findings that adversarial training helps improve input-robustness, we theoretically show that adversarially trained models have converged excess risk on OOD data, and empirically verify it on both image classification and natural language understanding tasks. Besides, in the paradigm of first pre-training and then fine-tuning, we theoretically show that a pre-trained model that is more robust to input perturbation provides a better initialization for generalization on downstream OOD data. Empirically, after fine-tuning, this better-initialized model from adversarial pre-training also has better OOD generalization.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 39 of its 62 references that are in the map. First posted on arXiv 24 May 2021.

Builds on

39 of 62 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  9. 2013Intriguing properties of neural networkscited by 5,155
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
27 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  3. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  7. 2014Convex Optimization: Algorithms and Complexitycited by 575
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  9. 2018Do Better ImageNet Models Transfer Better?cited by 565
  10. 2018Generalizing to Unseen Domains via Adversarial Data Augmentationcited by 451
  11. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 431
  12. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  13. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
  14. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  15. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
  16. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
  17. 2019Adversarial Examples Improve Image Recognitioncited by 286
  18. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 264
  19. 2018Adversarial Logit Pairingcited by 224
  20. 2019Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methodscited by 148
  21. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 134
  22. 2018Rademacher Complexity for Adversarially Robust Generalizationcited by 132
  23. 2019Adversarial Training Can Hurt Generalizationcited by 132
  24. 2020Unadversarial Examples: Designing Objects for Robust Visioncited by 45
  25. 2020Adversarially-Trained Deep Nets Transfer Better: Illustration on Image Classificationcited by 18
  26. 2019Inductive Bias of Gradient Descent based Adversarial Training on Separable Datacited by 12
  27. 2020Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performancecited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 18; 2023: 20; 2024: 12; 2025: 9; 2026: 3.

66 papers.

Sort:

By area:

  1. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  2. 2022Adversarial Training for High-Stakes Reliabilitycited by 60
  3. 2022ZIN: When and How to Learn Invariance Without Environment Partition?cited by 58
  4. 2024A Survey of Graph Neural Networks in Real world: Imbalance, Noise, Privacy and OOD Challengescited by 54
  5. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
  6. 2022Out-of-distribution Generalization with Causal Invariant Transformationscited by 45
  7. 2023On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codexcited by 41
  8. 2021PRIME: A few primitives can boost robustness to common corruptionscited by 40
  9. 2022SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detectioncited by 33
  10. 2023Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferabilitycited by 32
  11. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  12. 2022Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustnesscited by 23
38 more
  1. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  2. 2023Improving Generalization of Adversarial Training via Robust Critical Fine-Tuningcited by 23
  3. 2023A Survey on Out-of-Distribution Evaluation of Neural NLP Modelscited by 20
  4. 2022Improving Out-of-Distribution Generalization by Adversarial Training with Structured Priorscited by 17
  5. 2021An Educated Warm Start For Deep Image Prior-Based Micro CT Reconstructioncited by 16
  6. 2023Making Vision Transformers Truly Shift-Equivariantcited by 16
  7. 2025CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Surveycited by 15
  8. 2023Explore and Exploit the Diverse Knowledge in Model Zoo for Domain Generalizationcited by 14
  9. 2023Defenses in Adversarial Machine Learning: A Surveycited by 14
  10. 2024Smoothness Really Matters: A Simple Yet Effective Approach for Unsupervised Graph Domain Adaptationcited by 14
  11. 2022RODD: A Self-Supervised Approach for Robust Out-of-Distribution Detectioncited by 13
  12. 2022ZooD: Exploiting Model Zoo for Out-of-Distribution Generalizationcited by 12
  13. 2021Partial Domain Adaptation without Domain Alignmentcited by 11
  14. 2024FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusioncited by 11
  15. 2023Outlier Robust Adversarial Trainingcited by 10
  16. 2025Generalizable deep learning for photoplethysmography-based blood pressure estimation -- A Benchmarking Studycited by 9
  17. 2021NoiLIn: Improving Adversarial Training and Correcting Stereotype of Noisy Labelscited by 8
  18. 2022Breaking Correlation Shift via Conditional Invariant Regularizercited by 7
  19. 2023Revisiting Image Classifier Training for Improved Certified Robust Defense against Adversarial Patchescited by 6
  20. 2023Data Optimization in Deep Learning: A Surveycited by 6
  21. 2024Out-of-Distribution Data: An Acquaintance of Adversarial Examples -- A Surveycited by 6
  22. 2024Long-Tailed Out-of-Distribution Detection: Prioritizing Attention to Tailcited by 6
  23. 2022Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLUcited by 5
  24. 2022On the Connection between Invariant Learning and Adversarial Training for Out-of-Distribution Generalizationcited by 5
  25. 2023Can Large Language Models Change User Preference Adversarially?cited by 5
  26. 2023Improving Model Generalization by On-manifold Adversarial Augmentation in the Frequency Domaincited by 5
  27. 2023Generalizable Zero-Shot Speaker Adaptive Speech Synthesis with Disentangled Representationscited by 5
  28. 2023PerturbScore: Connecting Discrete and Continuous Perturbations in NLPcited by 5
  29. 2022Jacobian Norm for Unsupervised Source-Free Domain Adaptationcited by 4
  30. 2024LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Viewscited by 4
  31. 2024Language-Enhanced Latent Representations for Out-of-Distribution Detection in Autonomous Drivingcited by 4
  32. 2025Measure-Theoretic Anti-Causal Representation Learningcited by 4
  33. 2022Methods for Estimating and Improving Robustness of Language Modelscited by 3
  34. 2022Unveiling the Tapestry: the Interplay of Generalization and Forgetting in Continual Learningcited by 3
  35. 2025Improved Diffusion-based Generative Model with Better Adversarial Robustnesscited by 3
  36. 2026TANDEM: Bi-Level Data Mixture Optimization with Twin Networkscited by 3
  37. 2022Saliency Guided Adversarial Training for Learning Generalizable Features with Applications to Medical Imaging Classification Systemcited by 2
  38. 2025AdvSumm: Adversarial Training for Bias Mitigation in Text Summarizationcited by 2

50 most cited of 66 All 66 (JSON)