mlsanityData to 7 Oct 2026

Pretrained Transformers Do not Always Improve Robustness

arXiv 2210.07663 ↗First posted 14 Oct 2022cs.CL cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Pretrained Transformers (PT) have been shown to improve Out of Distribution (OOD) robustness than traditional models such as Bag of Words (BOW), LSTMs, Convolutional Neural Networks (CNN) powered by Word2Vec and Glove embeddings. How does the robustness comparison hold in a real world setting where some part of the dataset can be noisy? Do PT also provide more robust representation than traditional models on exposure to noisy data? We perform a comparative study on 10 models and find an empirical evidence that PT provide less robust representation than traditional models on exposure to noisy data. We investigate further and augment PT with an adversarial filtering (AF) mechanism that has been shown to improve OOD generalization. However, increase in generalization does not necessarily increase robustness, as we find that noisy data fools the AF method powered by PT.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

22 of 29 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  11. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  12. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
10 more
  1. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  2. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  3. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  4. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  5. 2020Adversarial Filters of Dataset Biasescited by 159
  6. 2015Towards Universal Paraphrastic Sentence Embeddingscited by 59
  7. 2021How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluationcited by 24
  8. 2020DQI: Measuring Data Quality in NLPcited by 23
  9. 2022Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustnesscited by 23
  10. 2020Our Evaluation Metric Needs an Update to Encourage Generalizationcited by 12

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 1.

2 papers.

  1. 2023Adventures of Trustworthy Vision-Language Models: A Surveycited by 10
  2. 2025Reasoning Bias of Next Token Prediction Trainingcited by 3