# Improving the Out-Of-Distribution Generalization Capability of Language Models: Counterfactually-Augmented Data is not Enough

arXiv: https://arxiv.org/abs/2302.09345 · HTML: https://mlsanity.com/paper/2302.09345/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 18 Feb 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [The Risks of Invariant Risk Minimization](https://mlsanity.com/paper/2010.05761/) (2020)
- [Robustness to Spurious Correlations in Text Classification via Automatically Generated Counterfactuals](https://mlsanity.com/paper/2012.10040/) (2020)
- [Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision](https://mlsanity.com/paper/2004.09034/) (2020)
- [Identifying Spurious Correlations for Robust Text Classification](https://mlsanity.com/paper/2010.02458/) (2020)
- [An Investigation of the (In)effectiveness of Counterfactually Augmented Data](https://mlsanity.com/paper/2107.00753/) (2021)
- [Exploring the Efficacy of Automatically Generated Counterfactuals for Sentiment Analysis](https://mlsanity.com/paper/2106.15231/) (2021)
- [Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data](https://mlsanity.com/paper/2010.04762/) (2020)
- [Explicit Interaction Model towards Text Classification](https://mlsanity.com/paper/1811.09386/) (2018)

## Built on by (most cited first, 3 of 3)

- [Comparable Demonstrations are Important in In-Context Learning: A Novel Perspective on Demonstration Selection](https://mlsanity.com/paper/2312.07476/) (2023)
- [Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Models](https://mlsanity.com/paper/2510.18526/) (2025)
- [Chain-of-Thought Tuning: Masked Language Models can also Think Step By Step in Natural Language Understanding](https://mlsanity.com/paper/2310.11721/) (2023)

## Abstract (arXiv)

Counterfactually-Augmented Data (CAD) has the potential to improve language models' Out-Of-Distribution (OOD) generalization capability, as CAD induces language models to exploit causal features and exclude spurious correlations. However, the empirical results of OOD generalization on CAD are not as efficient as expected. In this paper, we attribute the inefficiency to Myopia Phenomenon caused by CAD: language models only focus on causal features that are edited in the augmentation and exclude other non-edited causal features. As a result, the potential of CAD is not fully exploited. Based on the structural properties of CAD, we design two additional constraints to help language models extract more complete causal features contained in CAD, thus improving the OOD generalization capability. We evaluate our method on two tasks: Sentiment Analysis and Natural Language Inference, and the experimental results demonstrate that our method could unlock CAD's potential and improve language models' OOD generalization capability.