Data to 14 Oct 2026

An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models

arXiv 2007.06778 ↗First posted 14 Jul 2020cs.CL cs.LGcited by 134

Not on the map. Why?

Abstract (arXiv)

Recent work has shown that pre-trained language models such as BERT improve robustness to spurious correlations in the dataset. Intrigued by these results, we find that the key to their success is generalization from a small amount of counterexamples where the spurious correlations do not hold. When such minority examples are scarce, pre-trained models perform as poorly as models trained from scratch. In the case of extreme minority, we propose to use multi-task learning (MTL) to improve generalization. Our experiments on natural language inference and paraphrase identification show that MTL with the right auxiliary tasks significantly improves performance on challenging examples without hurting the in-distribution performance. Further, we show that the gain from MTL mainly comes from improved generalization from the minority examples. Our results highlight the importance of data diversity for overcoming spurious correlations.

Key facts. Cited by 134 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 31; 2022: 32; 2023: 34; 2024: 17; 2025: 11; 2026: 9.

134 papers.

Sort:

By area:

2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 109
2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 97
2021Towards Principled Disentanglement for Domain Generalizationcited by 91
2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 86
2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 85
2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78

38 more

2021OOD-GNN: Out-of-Distribution Generalized Graph Neural Networkcited by 78
2021Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Modelscited by 76
2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
2021Improved OOD Generalization via Adversarial Training and Pre-trainingcited by 66
2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49
2023Explore Spurious Correlations at the Concept Level in Language Models for Text Classificationcited by 48
2021An Investigation of the (In)effectiveness of Counterfactually Augmented Datacited by 42
2022Are Vision Transformers Robust to Spurious Correlations?cited by 36
2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
2021Memorisation versus Generalisation in Pre-trained Language Modelscited by 32
2021HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizationcited by 32
2021On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Studycited by 31
2022ModelDiff: A Framework for Comparing Learning Algorithmscited by 30
2021Supervising Model Attention with Human Explanations for Robust Natural Language Inferencecited by 29
2023Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Modelscited by 29
2022Are All Spurious Features in Natural Language Alike? An Analysis through a Causal Lenscited by 28
2021Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematicscited by 27
2022A Rationale-Centric Framework for Human-in-the-loop Machine Learningcited by 27
2021Benchmarking Multimodal AutoML for Tabular Data with Text Fieldscited by 26
2022Exploring The Landscape of Distributional Robustness for Question Answering Modelscited by 23
2021Interpreting the Robustness of Neural NLP Models to Textual Perturbationscited by 17
2021On the Copying Behaviors of Pre-Training for Neural Machine Translationcited by 15
2021Invariant Language Modelingcited by 15
2025Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associationscited by 15
2023Distributionally Robust Optimization with Probabilistic Groupcited by 14
2024Gumbel Counterfactual Generation From Language Modelscited by 14
2021Empowering Language Understanding with Counterfactual Reasoningcited by 13
2021Robustifying Multi-hop QA through Pseudo-Evidentiality Trainingcited by 13
2022Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentationcited by 13
2021Robustness Challenges in Model Distillation and Pruning for Natural Language Understandingcited by 12
2022Measures of Information Reflect Memorization Patternscited by 12
2024Tri-Level Navigator: LLM-Empowered Tri-Level Learning for Time Series OOD Generalizationcited by 12
2024What makes a good metric? Evaluating automatic metrics for text-to-image consistencycited by 12
2022Robust Natural Language Processing: Recent Advances, Challenges, and Future Directionscited by 11
2022Kernel-Whitening: Overcome Dataset Bias with Isotropic Sentence Embeddingcited by 11
2023Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysiscited by 11

50 most cited of 134 All 134 (JSON)