# Causality can systematically address the monsters under the bench(marks)

arXiv: https://arxiv.org/abs/2502.05085 · HTML: https://mlsanity.com/paper/2502.05085/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2025.

## Built on by (most cited first, 4 of 4)

- [Causality is Key for Interpretability Claims to Generalise](https://mlsanity.com/paper/2602.16698/) (2026)
- [Are Language Models Efficient Reasoners? A Perspective from Logic Programming](https://mlsanity.com/paper/2510.25626/) (2025)
- [Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI](https://mlsanity.com/paper/2603.25033/) (2026)
- [An Investigation of the NeurIPS and ICML 2025 Position Tracks](https://mlsanity.com/paper/2608.16894/) (2026)

## Abstract (arXiv)

Effective and reliable evaluation is essential for advancing empirical machine learning. However, the increasing accessibility of generalist models and the progress towards ever more complex, high-level tasks make systematic evaluation more challenging. Benchmarks are plagued by various biases, artifacts, or leakage, while models may behave unreliably due to poorly explored failure modes. Haphazard treatments and inconsistent formulations of such "monsters" can contribute to a duplication of efforts, a lack of trust in results, and unsupported inferences. In this position paper, we argue causality offers an ideal framework to systematically address these challenges. By making causal assumptions in an approach explicit, we can faithfully model phenomena, formulate testable hypotheses with explanatory power, and leverage principled tools for analysis. To make causal model design more accessible, we identify several useful Common Abstract Topologies (CATs) in causal graphs which help gain insight into the reasoning abilities in large language models. Through a series of case studies, we demonstrate how the precise yet pragmatic language of causality clarifies the strengths and limitations of a method and inspires new approaches for systematic progress.