# Compositional Causal Reasoning Evaluation in Language Models

arXiv: https://arxiv.org/abs/2503.04556 · HTML: https://mlsanity.com/paper/2503.04556/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning.
- Builds on 17 of its 113 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Placed in

- LLM reasoning / [Causal reasoning in LLMs](https://mlsanity.com/lr/causal-reasoning-in-llms/)

## Builds on (most cited first, 17 of 17)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering](https://mlsanity.com/paper/2402.07630/) (2024)
- [OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data](https://mlsanity.com/paper/2410.01560/) (2024)
- [Desiderata for Representation Learning: A Causal Perspective](https://mlsanity.com/paper/2109.03795/) (2021)
- [COLA: A Benchmark for Compositional Text-to-image Retrieval](https://mlsanity.com/paper/2305.03689/) (2023)
- [C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset](https://mlsanity.com/paper/1704.08243/) (2017)
- [On Learning Necessary and Sufficient Causal Graphs](https://mlsanity.com/paper/2301.12389/) (2023)
- [Reasoning Elicitation in Language Models via Counterfactual Feedback](https://mlsanity.com/paper/2410.03767/) (2024)
- [Bounding probabilities of causation through the causal marginal problem](https://mlsanity.com/paper/2304.02023/) (2023)
- [Tightening Bounds on Probabilities of Causation By Merging Datasets](https://mlsanity.com/paper/2310.08406/) (2023)

## Built on by (most cited first, 6 of 6)

- [What-If World: A Causal Benchmark for General World Models in Embodied Scenarios](https://mlsanity.com/paper/2605.27589/) (2026)
- [CausalARC: Abstract Reasoning with Causal World Models](https://mlsanity.com/paper/2509.03636/) (2025)
- [CauSim: Scaling Causal Reasoning with Increasingly Complex Causal Simulators](https://mlsanity.com/paper/2605.09079/) (2026)
- [Why Prompt Optimization Works, and Why It Sometimes Doesn't: A Causal-Inspired Edit-Level Analysis](https://mlsanity.com/paper/2605.26655/) (2026)
- [Operadic consistency: a label-free signal for compositional reasoning failures in LLMs](https://mlsanity.com/paper/2606.13649/) (2026)
- [Position: Reasoning is a Learnable Rule-Based Process](https://mlsanity.com/paper/2608.12325/) (2026)

## Abstract (arXiv)

Causal reasoning and compositional reasoning are two core aspirations in AI. Measuring the extent of these behaviors requires principled evaluation methods. We explore a unified perspective that considers both behaviors simultaneously, termed compositional causal reasoning (CCR): the ability to infer how causal measures compose and, equivalently, how causal quantities propagate through graphs. We instantiate a framework for the systematic evaluation of CCR for the average treatment effect and the probability of necessity and sufficiency. As proof of concept, we demonstrate CCR evaluation for language models in the LLama, Phi, and GPT families. On a math word problem, our framework revealed a range of taxonomically distinct error patterns. CCR errors increased with the complexity of causal paths for all models except o1.