# EconCausal: A Context-Aware Economic Reasoning Benchmark for Large Language Models

arXiv: https://arxiv.org/abs/2510.07231 · HTML: https://mlsanity.com/paper/2510.07231/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning.
- Builds on 5 of its 25 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Placed in

- LLM reasoning / [Causal reasoning in LLMs](https://mlsanity.com/lr/causal-reasoning-in-llms/)

## Builds on (most cited first, 5 of 5)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Causal Claims in Economics](https://mlsanity.com/paper/2501.06873/) (2025)

## Built on by (most cited first, 3 of 3)

- [CausalDS: Benchmarking Causal Reasoning in Data-Science Agents](https://mlsanity.com/paper/2607.08093/) (2026)
- [Ideological Bias in LLMs' Economic Causal Reasoning](https://mlsanity.com/paper/2604.21334/) (2026)
- [Vernier: Probing Representational Misalignment Behind Lexical Gaps in Causal Reasoning](https://mlsanity.com/paper/2606.15733/) (2026)

## Abstract (arXiv)

Socio-economic causal effects depend heavily on their institutional and environmental contexts. The same intervention can produce different, even opposite, effects across regulatory regimes, market conditions, time periods, or populations. This poses a challenge for large language models (LLMs) in decision-support roles: can they infer the direction of a causal effect under a specified context, and revise that judgment when the context changes? To address this, we introduce EconCausal, a large-scale benchmark of 10,490 context-annotated causal triplets extracted from 2,595 high-quality empirical studies in top-tier economics and finance journals, constructed through a rigorous four-stage pipeline with multi-run consensus, context refinement, and multi-critic filtering. Across 19 models, we observe source-sign over-transfer in cross-context prediction. While top models reach ~84% accuracy in single-context prediction, mean accuracy is 74.75% overall but drops to 44.27% when source and target signs disagree. On these cases, accuracy is 11.54 percentage points (pp) lower with the signed source example than with the target input alone. Mean accuracy is below 60% under corrupted source signs. Models also struggle to predict study-reported nonsignificant effects (19.16% accuracy). The dataset and benchmark are publicly available at https://anonymous.4open.science/r/econcausal-benchmark-6F12.