mlsanityData to 14 Oct 2026

CLEAR: Can Language Models Really Understand Causal Graphs?

arXiv 2406.16605 ↗First posted 24 Jun 2024cs.CL cs.AI cs.LGcited by 14

On the map: LLM reasoning → Causal reasoning in LLMs

Abstract (arXiv)

Causal reasoning is a cornerstone of how humans interpret the world. To model and reason about causality, causal graphs offer a concise yet effective solution. Given the impressive advancements in language models, a crucial question arises: can they really understand causal graphs? To this end, we pioneer an investigation into language models' understanding of causal graphs. Specifically, we develop a framework to define causal graph understanding, by assessing language models' behaviors through four practical criteria derived from diverse disciplines (e.g., philosophy and psychology). We then develop CLEAR, a novel benchmark that defines three complexity levels and encompasses 20 causal graph-based tasks across these levels. Finally, based on our framework and benchmark, we conduct extensive experiments on six leading language models and summarize five empirical findings. Our results indicate that while language models demonstrate a preliminary understanding of causal graphs, significant potential for improvement remains. Our project website is at https://github.com/OpenCausaLab/CLEAR.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning. Builds on 36 of its 77 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

36 of 77 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data27 Feb
  2. 2024Robust agents learn causal world models16 Feb
  3. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causality28 Apr

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  9. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  10. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  11. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  12. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
21 more
  1. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 274
  2. 2023Language is All a Graph Needscited by 220
  3. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  4. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  5. 2023Can Large Language Models Infer Causation from Correlation?cited by 146
  6. 2019Noisier2Noise: Learning to Denoise from Unpaired Noisy Datacited by 126
  7. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 119
  8. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  9. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmarkcited by 63
  10. 2024GraphInstruct: Empowering Large Language Models with Graph Understanding and Reasoning Capabilitycited by 58
  11. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  12. 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Taskscited by 44
  13. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  14. 2016The AGI Containment Problemcited by 36
  15. 2023Using Captum to Explain Generative Language Modelscited by 33
  16. 2023Beyond Text: A Deep Dive into Large Language Models' Ability on Understanding Graph Datacited by 25
  17. 2024Cause and Effect: Can Large Language Models Truly Understand Causality?cited by 23
  18. 2024"A good pun is its own reword": Can Large Language Models Understand Puns?cited by 21
  19. 2023Graph Meets LLMs: Towards Large Graph Modelscited by 20
  20. 2022Machine Reading, Fast and Slow: When Do Models "Understand" Language?cited by 13
  21. 2024Learning Beyond Pattern Matching? Assaying Mathematical Understanding in LLMscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 4.

14 papers.

Sort:

By area:

  1. 2024CausalGraph2LLM: Evaluating LLMs for Causal Queriescited by 12
  2. 2024Beyond Graphs: Can Large Language Models Comprehend Hypergraphs?cited by 11
  3. 2025A Survey on Enhancing Causal Reasoning Ability of Large Language Modelscited by 8
  4. 2025Causal-aware Large Language Models: Enhancing Decision-Making Through Learning, Adapting and Actingcited by 6
  5. 2025Realizing LLMs' Causal Potential Requires Science-Grounded, Novel Benchmarkscited by 4
  6. 2026CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientistscited by 4
  7. 2025IGDA: Interactive Graph Discovery through Large Language Model Agentscited by 3
  8. 2024Do LLMs Have the Generalization Ability in Conducting Causal Inference?cited by 2
  9. 2024CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Modelscited by 2
  10. 2025IP-Dialog: Evaluating Implicit Personalization in Dialogue Systems with Synthetic Datacited by 1
  11. 2025Causal Cartographer: From Mapping to Reasoning Over Counterfactual Worldscited by 0
  12. 2026DAGverse: Building Document-Grounded Semantic DAGs from Scientific Paperscited by 0
2 more
  1. 2026Learning to Deny: Action Denial in Multimodal Large Language Modelscited by 0
  2. 2026CausalVerify: End-to-End Verification of Causal Analyses by Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CLEAR-3K: Assessing Causal Explanatory Capabilities in Language Models
  2. 2024Causal Evaluation of Language Models
  3. 2023CLadder: Assessing Causal Reasoning in Language Models
  4. 2024A Critical Review of Causal Reasoning Benchmarks for Large Language Models
  5. 2024OCDB: Revisiting Causal Discovery with a Comprehensive Benchmark and Evaluation Framework
  6. 2023Causal models in string diagrams
  7. 2025Causality can systematically address the monsters under the bench(marks)
  8. 2025Can Large Language Models Infer Causal Relationships from Real-World Text?