# CausalGraph2LLM: Evaluating LLMs for Causal Queries

arXiv: https://arxiv.org/abs/2410.15939 · HTML: https://mlsanity.com/paper/2410.15939/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Differentiable Causal Discovery from Interventional Data](https://mlsanity.com/paper/2007.01754/) (2020)
- [Causal Discovery from Heterogeneous/Nonstationary Data with Independent Changes](https://mlsanity.com/paper/1903.01672/) (2019)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [Hypothesis Search: Inductive Reasoning with Language Models](https://mlsanity.com/paper/2309.05660/) (2023)
- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [Can large language models build causal graphs?](https://mlsanity.com/paper/2303.05279/) (2023)
- [Integrating Large Language Model for Improved Causal Discovery](https://mlsanity.com/paper/2306.16902/) (2023)
- [Causal Order: The Key to Leveraging Imperfect Experts in Causal Inference](https://mlsanity.com/paper/2310.15117/) (2023)
- [The impact of prior knowledge on causal structure learning](https://mlsanity.com/paper/2102.00473/) (2021)
- [Enhancing Causal Reasoning in Large Language Models: A Causal Attribution Model for Precision Fine-Tuning](https://mlsanity.com/paper/2401.00139/) (2023)
- [CLEAR: Can Language Models Really Understand Causal Graphs?](https://mlsanity.com/paper/2406.16605/) (2024)
- [Your Large Language Model is Secretly a Fairness Proponent and You Should Prompt it Like One](https://mlsanity.com/paper/2402.12150/) (2024)
- [UniCausal: Unified Benchmark and Repository for Causal Text Mining](https://mlsanity.com/paper/2208.09163/) (2022)
- [Large Language Models are biased to overestimate profoundness](https://mlsanity.com/paper/2310.14422/) (2023)
- [Context-Aware Reasoning On Parametric Knowledge for Inferring Causal Variables](https://mlsanity.com/paper/2409.02604/) (2024)

## Built on by (most cited first, 12 of 12)

- [CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation](https://mlsanity.com/paper/2602.20571/) (2026)
- [Revealing Multimodal Causality with Large Language Models](https://mlsanity.com/paper/2509.17784/) (2025)
- [Causally-Enhanced Reinforcement Policy Optimization](https://mlsanity.com/paper/2509.23095/) (2025)
- [Toward Causal-Visual Programming: Enhancing Agentic Reasoning in Low-Code Environments](https://mlsanity.com/paper/2509.25282/) (2025)
- [ClimateCause: Complex and Implicit Causal Structures in Climate Reports](https://mlsanity.com/paper/2604.14856/) (2026)
- [Why LLMs Fail at Causal Discovery and How Interventional Agents Escape](https://mlsanity.com/paper/2605.27567/) (2026)
- [CausalDS: Benchmarking Causal Reasoning in Data-Science Agents](https://mlsanity.com/paper/2607.08093/) (2026)
- [Transformers Can Learn Connectivity in Some Graphs but Not Others](https://mlsanity.com/paper/2509.22343/) (2025)
- [Causal-Enhanced AI Agents for Medical Research Screening](https://mlsanity.com/paper/2601.02814/) (2026)
- [DAGverse: Building Document-Grounded Semantic DAGs from Scientific Papers](https://mlsanity.com/paper/2603.25293/) (2026)
- [Evaluating Bivariate Causal Statements Based on Mutual Compatibility](https://mlsanity.com/paper/2606.00278/) (2026)
- [CausalSteward: An Agentic Divide-Conquer-Combine Copilot for Causal Discovery](https://mlsanity.com/paper/2607.01936/) (2026)

## Abstract (arXiv)

Causality is essential in scientific research, enabling researchers to interpret true relationships between variables. These causal relationships are often represented by causal graphs, which are directed acyclic graphs. With the recent advancements in Large Language Models (LLMs), there is an increasing interest in exploring their capabilities in causal reasoning and their potential use to hypothesize causal graphs. These tasks necessitate the LLMs to encode the causal graph effectively for subsequent downstream tasks. In this paper, we introduce CausalGraph2LLM, a comprehensive benchmark comprising over 700k queries across diverse causal graph settings to evaluate the causal reasoning capabilities of LLMs. We categorize the causal queries into two types: graph-level and node-level queries. We benchmark both open-sourced and propriety models for our study. Our findings reveal that while LLMs show promise in this domain, they are highly sensitive to the encoding used. Even capable models like GPT-4 and Gemini-1.5 exhibit sensitivity to encoding, with deviations of about 60%. We further demonstrate this sensitivity for downstream causal intervention tasks. Moreover, we observe that LLMs can often display biases when presented with contextual information about a causal graph, potentially stemming from their parametric memory.