# Unlock the Potential of Counterfactually-Augmented Data in Out-Of-Distribution Generalization

arXiv: https://arxiv.org/abs/2310.06666 · HTML: https://mlsanity.com/paper/2310.06666/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 69 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 30 of 37)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [The Risks of Invariant Risk Minimization](https://mlsanity.com/paper/2010.05761/) (2020)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [A Survey on Causal Inference](https://mlsanity.com/paper/2002.02770/) (2020)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [A Fine-Grained Analysis on Distribution Shift](https://mlsanity.com/paper/2110.11328/) (2021)
- [Understanding the Failure Modes of Out-of-Distribution Generalization](https://mlsanity.com/paper/2010.15775/) (2020)
- [Efficient Domain Generalization via Common-Specific Low-Rank Decomposition](https://mlsanity.com/paper/2003.12815/) (2020)
- [CausaLM: Causal Model Explanation Through Counterfactual Language Models](https://mlsanity.com/paper/2005.13407/) (2020)
- [DIVA: Domain Invariant Variational Autoencoders](https://mlsanity.com/paper/1905.10427/) (2019)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness](https://mlsanity.com/paper/2009.10195/) (2020)
- [Robustness to Spurious Correlations in Text Classification via Automatically Generated Counterfactuals](https://mlsanity.com/paper/2012.10040/) (2020)
- [AEDA: An Easier Data Augmentation Technique for Text Classification](https://mlsanity.com/paper/2108.13230/) (2021)
- [Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision](https://mlsanity.com/paper/2004.09034/) (2020)
- [Identifying Spurious Correlations for Robust Text Classification](https://mlsanity.com/paper/2010.02458/) (2020)
- [An Investigation of the (In)effectiveness of Counterfactually Augmented Data](https://mlsanity.com/paper/2107.00753/) (2021)

## Built on by (most cited first, 2 of 2)

- [Comparable Demonstrations are Important in In-Context Learning: A Novel Perspective on Demonstration Selection](https://mlsanity.com/paper/2312.07476/) (2023)
- [Chain-of-Thought Tuning: Masked Language Models can also Think Step By Step in Natural Language Understanding](https://mlsanity.com/paper/2310.11721/) (2023)

## Abstract (arXiv)

Counterfactually-Augmented Data (CAD) -- minimal editing of sentences to flip the corresponding labels -- has the potential to improve the Out-Of-Distribution (OOD) generalization capability of language models, as CAD induces language models to exploit domain-independent causal features and exclude spurious correlations. However, the empirical results of CAD's OOD generalization are not as efficient as anticipated. In this study, we attribute the inefficiency to the myopia phenomenon caused by CAD: language models only focus on causal features that are edited in the augmentation operation and exclude other non-edited causal features. Therefore, the potential of CAD is not fully exploited. To address this issue, we analyze the myopia phenomenon in feature space from the perspective of Fisher's Linear Discriminant, then we introduce two additional constraints based on CAD's structural properties (dataset-level and sentence-level) to help language models extract more complete causal features in CAD, thereby mitigating the myopia phenomenon and improving OOD generalization capability. We evaluate our method on two tasks: Sentiment Analysis and Natural Language Inference, and the experimental results demonstrate that our method could unlock the potential of CAD and improve the OOD generalization performance of language models by 1.0% to 5.9%.