# METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models

arXiv: https://arxiv.org/abs/2604.11502 · HTML: https://mlsanity.com/paper/2604.11502/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning.
- Builds on 8 of its 47 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Placed in

- LLM reasoning / [Causal reasoning in LLMs](https://mlsanity.com/lr/causal-reasoning-in-llms/)

## Builds on (most cited first, 8 of 8)

- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [From Understanding to Utilization: A Survey on Explainability for Large Language Models](https://mlsanity.com/paper/2401.12874/) (2024)
- [Generative AI for Synthetic Data Generation: Methods, Challenges and the Future](https://mlsanity.com/paper/2403.04190/) (2024)
- [Causal Evaluation of Language Models](https://mlsanity.com/paper/2405.00622/) (2024)
- [A Critical Review of Causal Reasoning Benchmarks for Large Language Models](https://mlsanity.com/paper/2407.08029/) (2024)
- [Can Large Language Models Infer Causal Relationships from Real-World Text?](https://mlsanity.com/paper/2505.18931/) (2025)

## Built on by (most cited first, 1 of 1)

- [Vernier: Probing Representational Misalignment Behind Lexical Gaps in Causal Reasoning](https://mlsanity.com/paper/2606.15733/) (2026)

## Abstract (arXiv)

Contextual causal reasoning is a critical yet challenging capability for Large Language Models (LLMs). Existing benchmarks, however, often evaluate this skill in fragmented settings, failing to ensure context consistency or cover the full causal hierarchy. To address this, we pioneer METER to systematically benchmark LLMs across all three levels of the causal ladder under a unified context setting. Our extensive evaluation of various LLMs reveals a significant decline in proficiency as tasks ascend the causal hierarchy. To diagnose this degradation, we conduct a deep mechanistic analysis via both error pattern identification and internal information flow tracing. Our analysis reveals two primary failure modes: (1) LLMs are susceptible to distraction by causally irrelevant but factually correct information at lower level of causality; and (2) as tasks ascend the causal hierarchy, faithfulness to the provided context degrades, leading to a reduced performance. We belive our work advances our understanding of the mechanisms behind LLM contextual causal reasoning and establishes a critical foundation for future research. Our code and dataset are available at https://github.com/SCUNLP/METER .