mlsanityData to 14 Oct 2026

Using Pre-Training Can Improve Model Robustness and Uncertainty

arXiv 1901.09960 ↗First posted 28 Jan 2019cs.LG cs.CV stat.MLcited by 373

Not on the map. Why?

Abstract (arXiv)

He et al. (2018) have called into question the utility of pre-training by showing that training from scratch can often yield similar performance to pre-training. We show that although pre-training may not improve performance on traditional classification metrics, it improves model robustness and uncertainty estimates. Through extensive experiments on adversarial examples, label corruption, class imbalance, out-of-distribution detection, and confidence calibration, we demonstrate large gains from pre-training and complementary effects with task-specific methods. We introduce adversarial pre-training and show approximately a 10% absolute improvement over the previous state-of-the-art in adversarial robustness. In some cases, using pre-training without task-specific methods also surpasses the state-of-the-art, highlighting the need for pre-training when evaluating future methods on robustness and uncertainty tasks.

Key facts. Cited by 373 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 115; 2022: 73; 2023: 77; 2024: 56; 2025: 34; 2026: 18.

373 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  2. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  3. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  4. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  5. 2021Deep Long-Tailed Learning: A Surveycited by 417
  6. 2022Post-hoc Concept Bottleneck Modelscited by 293
  7. 2021Unsolved Problems in ML Safetycited by 292
  8. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  9. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 279
  10. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 208
  11. 2021A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detectioncited by 207
  12. 2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 204
38 more
  1. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 195
  2. 2021Data Augmentation Can Improve Robustnesscited by 194
  3. 2021The Values Encoded in Machine Learning Researchcited by 172
  4. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  5. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 133
  6. 2021PixMix: Dreamlike Pictures Comprehensively Improve Safety Measurescited by 131
  7. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  8. 20223D Common Corruptions and Data Augmentationcited by 119
  9. 2022Robustness and Accuracy Could Be Reconcilable by (Proper) Definitioncited by 111
  10. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 109
  11. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 108
  12. 2021On the Adversarial Robustness of Vision Transformerscited by 106
  13. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  14. 2021Learning with Feature-Dependent Label Noise: A Progressive Approachcited by 102
  15. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  16. 2021Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attackscited by 101
  17. 2021AugMax: Adversarial Composition of Random Augmentations for Robust Trainingcited by 90
  18. 2022A Comparative Survey of Deep Active Learningcited by 89
  19. 2023Differentially Private Diffusion Models Generate Useful Synthetic Imagescited by 86
  20. 2023A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinkingcited by 85
  21. 2023Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Modelscited by 79
  22. 2021TorchGeo: Deep Learning With Geospatial Datacited by 73
  23. 2021On the effectiveness of adversarial training against common corruptionscited by 71
  24. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  25. 2021Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditionscited by 68
  26. 2021FREE: Feature Refinement for Generalized Zero-Shot Learningcited by 67
  27. 2021Improved OOD Generalization via Adversarial Training and Pre-trainingcited by 66
  28. 2023How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defensescited by 66
  29. 2021Understanding Softmax Confidence and Uncertaintycited by 63
  30. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 61
  31. 2022X-Risk Analysis for AI Researchcited by 60
  32. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  33. 2021Exploring Memorization in Adversarial Trainingcited by 55
  34. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 55
  35. 2021Convolutional Dynamic Alignment Networks for Interpretable Classificationscited by 54
  36. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  37. 2021Adversarial Robustness under Long-Tailed Distributioncited by 50
  38. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 50

50 most cited of 373 All 373 (JSON)