mlsanityData to 7 Oct 2026

Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond

arXiv 2109.00725 ↗First posted 2 Sep 2021cs.CL cs.LGcited by 177

Not on the map. Why?

Abstract (arXiv)

A fundamental goal of scientific research is to learn about causal relationships. However, despite its critical role in the life and social sciences, causality has not had the same importance in Natural Language Processing (NLP), which has traditionally placed more emphasis on predictive tasks. This distinction is beginning to fade, with an emerging area of interdisciplinary research at the convergence of causal inference and language processing. Still, research on causality in NLP remains scattered across domains without unified definitions, benchmark datasets and clear articulations of the challenges and opportunities in the application of causal inference to the textual domain, with its unique properties. In this survey, we consolidate research across academic areas and situate it in the broader NLP landscape. We introduce the statistical challenge of estimating causal effects with text, encompassing settings where text is used as an outcome, treatment, or to address confounding. In addition, we explore potential uses of causal inference to improve the robustness, fairness, and interpretability of NLP models. We thus provide a unified overview of causal inference for the NLP community.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Builds on 74 of its 127 references that are in the map. First posted on arXiv 2 Sep 2021.

Builds on

74 of 127 references.

References · 74

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  8. 2019Invariant Risk Minimizationcited by 1,832
  9. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  10. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  11. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  12. 2020In Search of Lost Domain Generalizationcited by 1,042
38 more
  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  2. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  3. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  4. 2019Attention is not Explanationcited by 916
  5. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  6. 2017Counterfactual Fairnesscited by 802
  7. 2014Distributed Representations of Sentences and Documentscited by 677
  8. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  9. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  10. 2013Domain Generalization via Invariant Feature Representationcited by 622
  11. 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 612
  12. 2015Estimation and Inference of Heterogeneous Treatment Effects using Random Forestscited by 597
  13. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  14. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  15. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  16. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  17. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  18. 2021Causal Abstractions of Neural Networkscited by 401
  19. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  20. 2020The Risks of Invariant Risk Minimizationcited by 262
  21. 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
  22. 2015Domain Generalization for Object Recognition with Multi-task Autoencoderscited by 253
  23. 2020Algorithmic Recourse: from Counterfactual Explanations to Interventionscited by 250
  24. 2019Causality for Machine Learningcited by 242
  25. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  26. 2019Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphologycited by 200
  27. 2017Avoiding Discrimination through Causal Reasoningcited by 191
  28. 2020Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AIcited by 166
  29. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 150
  30. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  31. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  32. 2021On Calibration and Out-of-domain Generalizationcited by 134
  33. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  34. 2019Towards a Critical Race Methodology in Algorithmic Fairnesscited by 124
  35. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  36. 2019It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitutioncited by 102
  37. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  38. 2020An Introduction to Proximal Causal Learningcited by 91

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 31; 2023: 47; 2024: 44; 2025: 27; 2026: 23.

177 papers.

Sort:

By area:

  1. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  2. 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 146
  3. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  4. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  5. 2023Can large language models build causal graphs?cited by 72
  6. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  7. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  8. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
  9. 2023Towards LLM-guided Causal Explainability for Black-box Text Classifierscited by 44
  10. 2023Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactualscited by 44
  11. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 44
  12. 2022How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysiscited by 42
38 more
  1. 2021Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspectivecited by 40
  2. 2023COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairscited by 39
  3. 2023Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Modelscited by 37
  4. 2022DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptationcited by 36
  5. 2023Prompting Large Language Models for Counterfactual Generation: An Empirical Studycited by 36
  6. 2023A Survey on Causal Discovery Methods for I.I.D. and Time Series Datacited by 35
  7. 2022Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal Viewcited by 31
  8. 2024Causal Inference with Large Language Model: A Surveycited by 31
  9. 2022FaiRR: Faithful and Robust Deductive Reasoning over Natural Languagecited by 29
  10. 2022DISCO: Distilling Counterfactuals with Large Language Modelscited by 29
  11. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 29
  12. 2023CRAB: Assessing the Strength of Causal Relationships Between Real-world Eventscited by 27
  13. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  14. 2023IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositionscited by 25
  15. 2022The Causal News Corpus: Annotating Causal Relations in Event Sentences from Newscited by 24
  16. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  17. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  18. 2022NeuroCounterfactuals: Beyond Minimal-Edit Counterfactuals for Richer Data Augmentationcited by 23
  19. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  20. 2023Counterfactual Learning on Graphs: A Surveycited by 22
  21. 2023COLA: Contextualized Commonsense Causal Reasoning from the Causal Inference Perspectivecited by 22
  22. 2023Causal Reinforcement Learning: A Surveycited by 22
  23. 2021Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLPcited by 21
  24. 2022An Empirical Study on Explanations in Out-of-Domain Settingscited by 21
  25. 2022Informativeness and Invariance: Two Perspectives on Spurious Correlations in Natural Languagecited by 21
  26. 2023A Causal View of Entity Bias in (Large) Language Modelscited by 21
  27. 2022Challenges in Applying Explainability Methods to Improve the Fairness of NLP Modelscited by 20
  28. 2022Affection: Learning Affective Explanations for Real-World Visual Datacited by 19
  29. 2024Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?cited by 19
  30. 2022Nuisances via Negativa: Adjusting for Spurious Correlations via Data Augmentationcited by 18
  31. 2024End-To-End Causal Effect Estimation from Unstructured Natural Language Datacited by 18
  32. 2023Towards Causal Foundation Model: on Duality between Causal Inference and Attentioncited by 17
  33. 2022ROCK: Causal Inference Principles for Reasoning about Commonsense Causalitycited by 16
  34. 2022Causal Intervention Improves Implicit Sentiment Analysiscited by 16
  35. 2024Causal Inference with Generative Artificial Intelligence: Application to Texts as Treatmentscited by 16
  36. 2021Natural Language Processing for Smart Healthcarecited by 14
  37. 2022Conceptualizing Treatment Leakage in Text-based Causal Inferencecited by 14
  38. 2022Domain Generalization -- A Causal Perspectivecited by 14

50 most cited of 177 All 177 (JSON)