# Large Language Models and Causal Inference in Collaboration: A Survey

arXiv: https://arxiv.org/abs/2403.09606 · HTML: https://mlsanity.com/paper/2403.09606/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 122 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 30 of 67)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Certified Robustness to Adversarial Word Substitutions](https://mlsanity.com/paper/1909.00986/) (2019)
- [Causal Parrots: Large Language Models May Talk Causality But Are Not Causal](https://mlsanity.com/paper/2308.13067/) (2023)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius](https://mlsanity.com/paper/2001.02378/) (2020)
- [Interpretability at Scale: Identifying Causal Mechanisms in Alpaca](https://mlsanity.com/paper/2305.08809/) (2023)
- [An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models](https://mlsanity.com/paper/2110.08527/) (2021)

## Built on by (most cited first, 26 of 26)

- [Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality](https://mlsanity.com/paper/2410.04780/) (2024)
- [Large Language Models for UAVs: Current State and Pathways to the Future](https://mlsanity.com/paper/2405.01745/) (2024)
- [AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models](https://mlsanity.com/paper/2406.10900/) (2024)
- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [Causality for Large Language Models](https://mlsanity.com/paper/2410.15319/) (2024)
- [CounterBench: Evaluating and Improving Counterfactual Reasoning in Large Language Models](https://mlsanity.com/paper/2502.11008/) (2025)
- [CausalEval: Towards Better Causal Reasoning in Language Models](https://mlsanity.com/paper/2410.16676/) (2024)
- [MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs](https://mlsanity.com/paper/2502.00997/) (2025)
- [Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios](https://mlsanity.com/paper/2406.05322/) (2024)
- [Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation](https://mlsanity.com/paper/2511.02303/) (2025)
- [CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstention](https://mlsanity.com/paper/2506.00519/) (2025)
- [The Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Education](https://mlsanity.com/paper/2404.02444/) (2024)
- [What Do Speech Foundation Models Not Learn About Speech?](https://mlsanity.com/paper/2410.12948/) (2024)
- [A Survey on Enhancing Causal Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2503.09326/) (2025)
- [Enhancing Answer Reliability Through Inter-Model Consensus of Large Language Models](https://mlsanity.com/paper/2411.16797/) (2024)
- [Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2406.13114/) (2024)
- [Multimodal Causal Reasoning Benchmark: Challenging Vision Large Language Models to Discern Causal Links Across Modalities](https://mlsanity.com/paper/2408.08105/) (2024)
- [Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identification](https://mlsanity.com/paper/2503.01896/) (2025)
- [Neutralizing Bias in LLM Reasoning using Entailment Graphs](https://mlsanity.com/paper/2503.11614/) (2025)
- [Do LLMs Have the Generalization Ability in Conducting Causal Inference?](https://mlsanity.com/paper/2410.11385/) (2024)
- [Generative Synthetic Data for Causal Inference: Pitfalls, Remedies, and Opportunities](https://mlsanity.com/paper/2604.23904/) (2026)
- [Unified Causality Analysis Based on the Degrees of Freedom](https://mlsanity.com/paper/2410.19469/) (2024)
- [A Technical Exploration of Causal Inference with Hybrid LLM Synthetic Data](https://mlsanity.com/paper/2511.00318/) (2025)
- [CRAwDAD: Causal Reasoning Augmentation with Dual-Agent Debate](https://mlsanity.com/paper/2511.22854/) (2025)
- [DeepFM-Crispr: Prediction of CRISPR On-Target Effects via Deep Learning](https://mlsanity.com/paper/2409.05938/) (2024)
- [CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehension](https://mlsanity.com/paper/2608.23172/) (2026)

## Abstract (arXiv)

Causal inference has shown potential in enhancing the predictive accuracy, fairness, robustness, and explainability of Natural Language Processing (NLP) models by capturing causal relationships among variables. The emergence of generative Large Language Models (LLMs) has significantly impacted various NLP domains, particularly through their advanced reasoning capabilities. This survey focuses on evaluating and improving LLMs from a causal view in the following areas: understanding and improving the LLMs' reasoning capacity, addressing fairness and safety issues in LLMs, complementing LLMs with explanations, and handling multimodality. Meanwhile, LLMs' strong reasoning capacities can in turn contribute to the field of causal inference by aiding causal relationship discovery and causal effect estimations. This review explores the interplay between causal inference frameworks and LLMs from both perspectives, emphasizing their collective potential to further the development of more advanced and equitable artificial intelligence systems.