mlsanityData to 7 Oct 2026

Pretrained Transformers Improve Out-of-Distribution Robustness

arXiv 2004.06100 ↗First posted 13 Apr 2020cs.CL cs.LGcited by 293

Not on the map. Why?

Abstract (arXiv)

Although pretrained Transformers such as BERT achieve high accuracy on in-distribution examples, do they generalize to new distributions? We systematically measure out-of-distribution (OOD) generalization for seven NLP datasets by constructing a new robustness benchmark with realistic distribution shifts. We measure the generalization of previous models including bag-of-words models, ConvNets, and LSTMs, and we show that pretrained Transformers' performance declines are substantially smaller. Pretrained transformers are also more effective at detecting anomalous or OOD examples, while many previous models are frequently worse than chance. We examine which factors affect robustness, finding that larger models are not necessarily more robust, distillation can be harmful, and more diverse pretraining data can enhance robustness. Finally, we show where future work can improve OOD robustness.

Key facts. Cited by 293 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 65; 2022: 73; 2023: 79; 2024: 35; 2025: 27; 2026: 14.

293 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  5. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  6. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  7. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  8. 2023ClimaX: A foundation model for weather and climatecited by 325
  9. 2022Large Language Models Are Reasoning Teacherscited by 318
  10. 2022Post-hoc Concept Bottleneck Modelscited by 289
  11. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  12. 2022Prompting GPT-3 To Be Reliablecited by 258
38 more
  1. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  2. 2023Pretraining Language Models with Human Preferencescited by 245
  3. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  4. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  5. 2023Large Language Models as General Pattern Machinescited by 196
  6. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  7. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  8. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  9. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  10. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 151
  11. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  12. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  13. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  14. 2022POEM: Out-of-Distribution Detection with Posterior Samplingcited by 108
  15. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  16. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  17. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  18. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  19. 2023A Comprehensive Survey on Source-free Domain Adaptationcited by 104
  20. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  21. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  22. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  23. 2021Pretrained Language Models for Text Generation: A Surveycited by 95
  24. 2021Towards Principled Disentanglement for Domain Generalizationcited by 91
  25. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  26. 2021Adversarially Adaptive Normalization for Single Domain Generalizationcited by 85
  27. 2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 85
  28. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 80
  29. 2021Types of Out-of-Distribution Texts and How to Detect Themcited by 80
  30. 2021RobustART: Benchmarking Robustness on Architecture Design and Training Techniquescited by 78
  31. 2021Standardized Max Logits: A Simple yet Effective Approach for Identifying Unexpected Road Obstacles in Urban-Scene Segmentationcited by 77
  32. 2021Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Modelscited by 76
  33. 2023Context-faithful Prompting for Large Language Modelscited by 76
  34. 2023Eight Things to Know about Large Language Modelscited by 73
  35. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  36. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  37. 2021Contrastive Out-of-Distribution Detection for Pretrained Transformerscited by 70
  38. 2021Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditionscited by 68

50 most cited of 293 All 293 (JSON)