# Empowering Language Understanding with Counterfactual Reasoning

arXiv: https://arxiv.org/abs/2106.03046 · HTML: https://mlsanity.com/paper/2106.03046/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 6 Jun 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Hierarchical Graph Representation Learning with Differentiable Pooling](https://mlsanity.com/paper/1806.08804/) (2018)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)
- [Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect](https://mlsanity.com/paper/2009.12991/) (2020)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [Addressing Failure Prediction by Learning Model Confidence](https://mlsanity.com/paper/1910.04851/) (2019)
- [Counterfactual Samples Synthesizing for Robust Visual Question Answering](https://mlsanity.com/paper/2003.06576/) (2020)
- [An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models](https://mlsanity.com/paper/2007.06778/) (2020)
- [Counterfactual Generative Networks](https://mlsanity.com/paper/2101.06046/) (2021)
- [Learning Independent Causal Mechanisms](https://mlsanity.com/paper/1712.00961/) (2017)
- [Entity-Relation Extraction as Multi-Turn Question Answering](https://mlsanity.com/paper/1905.05529/) (2019)
- [Counterfactual Zero-Shot and Open-Set Visual Recognition](https://mlsanity.com/paper/2103.00887/) (2021)
- [Coreference Resolution as Query-based Span Prediction](https://mlsanity.com/paper/1911.01746/) (2019)
- [Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance](https://mlsanity.com/paper/2005.00315/) (2020)
- [Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision](https://mlsanity.com/paper/2004.09034/) (2020)
- [Generating Plausible Counterfactual Explanations for Deep Transformers in Financial Text Classification](https://mlsanity.com/paper/2010.12512/) (2020)
- [Should Graph Convolution Trust Neighbors? A Simple Causal Inference Method](https://mlsanity.com/paper/2010.11797/) (2020)
- [Counterfactual Variable Control for Robust and Interpretable Question Answering](https://mlsanity.com/paper/2010.05581/) (2020)

## Built on by (most cited first, 13 of 13)

- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [A Rationale-Centric Framework for Human-in-the-loop Machine Learning](https://mlsanity.com/paper/2203.12918/) (2022)
- [Causal Reasoning of Entities and Events in Procedural Texts](https://mlsanity.com/paper/2301.10896/) (2023)
- [Uncovering Main Causalities for Long-tailed Information Extraction](https://mlsanity.com/paper/2109.05213/) (2021)
- [Boosting the Transferability of Video Adversarial Examples via Temporal Translation](https://mlsanity.com/paper/2110.09075/) (2021)
- [Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis](https://mlsanity.com/paper/2207.11652/) (2022)
- [Causal Intervention Improves Implicit Sentiment Analysis](https://mlsanity.com/paper/2208.09329/) (2022)
- [Causal Incremental Graph Convolution for Recommender System Retraining](https://mlsanity.com/paper/2108.06889/) (2021)
- [Causal Intervention for Abstractive Related Work Generation](https://mlsanity.com/paper/2305.13685/) (2023)
- [The Odyssey of Commonsense Causality: From Foundational Benchmarks to Cutting-Edge Reasoning](https://mlsanity.com/paper/2406.19307/) (2024)
- [On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Study](https://mlsanity.com/paper/2505.11839/) (2025)
- [Learning Robust Recommender from Noisy Implicit Feedback](https://mlsanity.com/paper/2112.01160/) (2021)
- [Do Not Harm Protected Groups in Debiasing Language Representation Models](https://mlsanity.com/paper/2310.18458/) (2023)

## Abstract (arXiv)

Present language understanding methods have demonstrated extraordinary ability of recognizing patterns in texts via machine learning. However, existing methods indiscriminately use the recognized patterns in the testing phase that is inherently different from us humans who have counterfactual thinking, e.g., to scrutinize for the hard testing samples. Inspired by this, we propose a Counterfactual Reasoning Model, which mimics the counterfactual thinking by learning from few counterfactual samples. In particular, we devise a generation module to generate representative counterfactual samples for each factual sample, and a retrospective module to retrospect the model prediction by comparing the counterfactual and factual samples. Extensive experiments on sentiment analysis (SA) and natural language inference (NLI) validate the effectiveness of our method.