# Can Foundation Models Talk Causality?

arXiv: https://arxiv.org/abs/2206.10591 · HTML: https://mlsanity.com/paper/2206.10591/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 44 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [Causal BERT : Language models for causality detection between events expressed in text](https://mlsanity.com/paper/2012.05453/) (2020)
- [Interventional Sum-Product Networks: Causal Inference with Tractable Probabilistic Models](https://mlsanity.com/paper/2102.10440/) (2021)
- [Artificial Intelligence is stupid and causal reasoning won't fix it](https://mlsanity.com/paper/2008.07371/) (2020)
- [Detecting non-causal artifacts in multivariate linear regression models](https://mlsanity.com/paper/1803.00810/) (2018)
- [The 30-Year Cycle In The AI Debate](https://mlsanity.com/paper/1810.04053/) (2018)

## Built on by (most cited first, 30 of 30)

- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey](https://mlsanity.com/paper/2404.01869/) (2024)
- [Can large language models build causal graphs?](https://mlsanity.com/paper/2303.05279/) (2023)
- [Causal Discovery with Language Models as Imperfect Experts](https://mlsanity.com/paper/2307.02390/) (2023)
- [Integrating Large Language Model for Improved Causal Discovery](https://mlsanity.com/paper/2306.16902/) (2023)
- [Large Language Models Are Not Strong Abstract Reasoners](https://mlsanity.com/paper/2305.19555/) (2023)
- [Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?](https://mlsanity.com/paper/2506.21215/) (2025)
- [The Essential Role of Causality in Foundation World Models for Embodied AI](https://mlsanity.com/paper/2402.06665/) (2024)
- [ALCM: Autonomous LLM-Augmented Causal Discovery Framework](https://mlsanity.com/paper/2405.01744/) (2024)
- [Large Language Models are Effective Priors for Causal Graph Discovery](https://mlsanity.com/paper/2405.13551/) (2024)
- [Causal Structure Learning Supervised by Large Language Model](https://mlsanity.com/paper/2311.11689/) (2023)
- [Masked Images Are Counterfactual Samples for Robust Fine-tuning](https://mlsanity.com/paper/2303.03052/) (2023)
- [Causal Evaluation of Language Models](https://mlsanity.com/paper/2405.00622/) (2024)
- [Position: Foundation Models Need Digital Twin Representations](https://mlsanity.com/paper/2505.03798/) (2025)
- [Towards a Benchmark for Causal Business Process Reasoning with LLMs](https://mlsanity.com/paper/2406.05506/) (2024)
- [Prompting Strategies for Enabling Large Language Models to Infer Causation from Correlation](https://mlsanity.com/paper/2412.13952/) (2024)
- [A Survey of Methods, Challenges and Perspectives in Causality](https://mlsanity.com/paper/2302.00293/) (2023)
- [Ice Cream Doesn't Cause Drowning: Benchmarking LLMs Against Statistical Pitfalls in Causal Inference](https://mlsanity.com/paper/2505.13770/) (2025)
- [Optimizing Data-driven Causal Discovery Using Knowledge-guided Search](https://mlsanity.com/paper/2304.05493/) (2023)
- [UPAR: A Kantian-Inspired Prompting Framework for Enhancing Large Language Model Capabilities](https://mlsanity.com/paper/2310.01441/) (2023)
- [Revealing Multimodal Causality with Large Language Models](https://mlsanity.com/paper/2509.17784/) (2025)
- [The Patient is not a Moving Document: A World Model Training Paradigm for Longitudinal EHR](https://mlsanity.com/paper/2601.22128/) (2026)
- [Regularized Multi-LLMs Collaboration for Enhanced Score-based Causal Discovery](https://mlsanity.com/paper/2411.17989/) (2024)
- [Leveraging Large Language Models for Automated Causal Loop Diagram Generation: Enhancing System Dynamics Modeling through Curated Prompting Techniques](https://mlsanity.com/paper/2503.21798/) (2025)
- [CARE: Turning LLMs Into Causal Reasoning Expert](https://mlsanity.com/paper/2511.16016/) (2025)
- [Sequential Causal Discovery with Noisy Language Model Priors](https://mlsanity.com/paper/2506.16234/) (2025)
- [Boosting Process-Correct CoT Reasoning by Modeling Solvability of Multiple-Choice QA](https://mlsanity.com/paper/2509.25941/) (2025)
- [DMCD: Semantic-Statistical Framework for Causal Discovery](https://mlsanity.com/paper/2602.20333/) (2026)
- [CausalSteward: An Agentic Divide-Conquer-Combine Copilot for Causal Discovery](https://mlsanity.com/paper/2607.01936/) (2026)
- [Externalized CPDAG Summaries Improve LLM Causal Deduction](https://mlsanity.com/paper/2609.31071/) (2026)

## Abstract (arXiv)

Foundation models are subject to an ongoing heated debate, leaving open the question of progress towards AGI and dividing the community into two camps: the ones who see the arguably impressive results as evidence to the scaling hypothesis, and the others who are worried about the lack of interpretability and reasoning capabilities. By investigating to which extent causal representations might be captured by these large scale language models, we make a humble efforts towards resolving the ongoing philosophical conflicts.