# CLEAR: Can Language Models Really Understand Causal Graphs?

arXiv: https://arxiv.org/abs/2406.16605 · HTML: https://mlsanity.com/paper/2406.16605/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning.
- Builds on 36 of its 77 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Placed in

- LLM reasoning / [Causal reasoning in LLMs](https://mlsanity.com/lr/causal-reasoning-in-llms/)

## Builds on (most cited first, 30 of 36)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs](https://mlsanity.com/paper/2307.03393/) (2023)
- [Language is All a Graph Needs](https://mlsanity.com/paper/2308.07134/) (2023)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering](https://mlsanity.com/paper/2212.10375/) (2022)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [Noisier2Noise: Learning to Denoise from Unpaired Noisy Data](https://mlsanity.com/paper/1910.11908/) (2019)
- [InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning](https://mlsanity.com/paper/2402.06332/) (2024)
- [Robust agents learn causal world models](https://mlsanity.com/paper/2402.10877/) (2024)
- [Can Large Language Models Understand Real-World Complex Instructions?](https://mlsanity.com/paper/2309.09150/) (2023)
- [Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark](https://mlsanity.com/paper/2305.14938/) (2023)
- [Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data](https://mlsanity.com/paper/2402.17644/) (2024)
- [GraphInstruct: Empowering Large Language Models with Graph Understanding and Reasoning Capability](https://mlsanity.com/paper/2403.04483/) (2024)
- [Data Scaling Laws in NMT: The Effect of Noise and Architecture](https://mlsanity.com/paper/2202.01994/) (2022)
- [MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks](https://mlsanity.com/paper/2310.19677/) (2023)
- [Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective](https://mlsanity.com/paper/2403.18346/) (2024)
- [The AGI Containment Problem](https://mlsanity.com/paper/1604.00545/) (2016)
- [Using Captum to Explain Generative Language Models](https://mlsanity.com/paper/2312.05491/) (2023)

## Built on by (most cited first, 14 of 14)

- [CausalGraph2LLM: Evaluating LLMs for Causal Queries](https://mlsanity.com/paper/2410.15939/) (2024)
- [Beyond Graphs: Can Large Language Models Comprehend Hypergraphs?](https://mlsanity.com/paper/2410.10083/) (2024)
- [A Survey on Enhancing Causal Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2503.09326/) (2025)
- [Causal-aware Large Language Models: Enhancing Decision-Making Through Learning, Adapting and Acting](https://mlsanity.com/paper/2505.24710/) (2025)
- [Realizing LLMs' Causal Potential Requires Science-Grounded, Novel Benchmarks](https://mlsanity.com/paper/2510.16530/) (2025)
- [CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists](https://mlsanity.com/paper/2605.26029/) (2026)
- [IGDA: Interactive Graph Discovery through Large Language Model Agents](https://mlsanity.com/paper/2502.17189/) (2025)
- [Do LLMs Have the Generalization Ability in Conducting Causal Inference?](https://mlsanity.com/paper/2410.11385/) (2024)
- [CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models](https://mlsanity.com/paper/2412.17970/) (2024)
- [IP-Dialog: Evaluating Implicit Personalization in Dialogue Systems with Synthetic Data](https://mlsanity.com/paper/2506.02449/) (2025)
- [Causal Cartographer: From Mapping to Reasoning Over Counterfactual Worlds](https://mlsanity.com/paper/2505.14396/) (2025)
- [DAGverse: Building Document-Grounded Semantic DAGs from Scientific Papers](https://mlsanity.com/paper/2603.25293/) (2026)
- [Learning to Deny: Action Denial in Multimodal Large Language Models](https://mlsanity.com/paper/2606.31187/) (2026)
- [CausalVerify: End-to-End Verification of Causal Analyses by Language Models](https://mlsanity.com/paper/2609.07944/) (2026)

## Abstract (arXiv)

Causal reasoning is a cornerstone of how humans interpret the world. To model and reason about causality, causal graphs offer a concise yet effective solution. Given the impressive advancements in language models, a crucial question arises: can they really understand causal graphs? To this end, we pioneer an investigation into language models' understanding of causal graphs. Specifically, we develop a framework to define causal graph understanding, by assessing language models' behaviors through four practical criteria derived from diverse disciplines (e.g., philosophy and psychology). We then develop CLEAR, a novel benchmark that defines three complexity levels and encompasses 20 causal graph-based tasks across these levels. Finally, based on our framework and benchmark, we conduct extensive experiments on six leading language models and summarize five empirical findings. Our results indicate that while language models demonstrate a preliminary understanding of causal graphs, significant potential for improvement remains. Our project website is at https://github.com/OpenCausaLab/CLEAR.