# Consistency evaluation of benchmarks used for causal discovery

arXiv: https://arxiv.org/abs/2606.01789 · HTML: https://mlsanity.com/paper/2606.01789/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 144 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Builds on (most cited first, 30 of 61)

- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [DAGs with NO TEARS: Continuous Optimization for Structure Learning](https://mlsanity.com/paper/1803.01422/) (2018)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Detecting causal associations in large nonlinear time series datasets](https://mlsanity.com/paper/1702.07007/) (2017)
- [Learning Bayesian Networks: The Combination of Knowledge and Statistical Data](https://mlsanity.com/paper/1302.6815/) (2013)
- [Score matching enables causal discovery of nonlinear additive noise models](https://mlsanity.com/paper/2203.04413/) (2022)
- [Efficient Neural Causal Discovery without Acyclicity Constraints](https://mlsanity.com/paper/2107.10483/) (2021)
- [Can large language models build causal graphs?](https://mlsanity.com/paper/2303.05279/) (2023)
- [BCD Nets: Scalable Variational Approaches for Bayesian Causal Discovery](https://mlsanity.com/paper/2112.02761/) (2021)
- [Ordering-Based Causal Discovery with Reinforcement Learning](https://mlsanity.com/paper/2105.06631/) (2021)
- [Causal Discovery in Heterogeneous Environments Under the Sparse Mechanism Shift Hypothesis](https://mlsanity.com/paper/2206.02013/) (2022)
- [Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning](https://mlsanity.com/paper/2107.00848/) (2021)
- [Causal Discovery Toolbox: Uncover causal relationships in Python](https://mlsanity.com/paper/1903.02278/) (2019)
- [Causal Order: The Key to Leveraging Imperfect Experts in Causal Inference](https://mlsanity.com/paper/2310.15117/) (2023)
- [Distinguishing cause from effect using observational data: methods and benchmarks](https://mlsanity.com/paper/1412.3773/) (2014)
- [BayesDAG: Gradient-Based Posterior Inference for Causal Discovery](https://mlsanity.com/paper/2307.13917/) (2023)
- [A Versatile Causal Discovery Framework to Allow Causally-Related Hidden Variables](https://mlsanity.com/paper/2312.11001/) (2023)
- [Neuropathic Pain Diagnosis Simulator for Causal Discovery Algorithm Evaluation](https://mlsanity.com/paper/1906.01732/) (2019)
- [Integrating Large Language Models in Causal Discovery: A Statistical Causal Approach](https://mlsanity.com/paper/2402.01454/) (2024)
- [The Case for Evaluating Causal Models Using Interventional Measures and Empirical Data](https://mlsanity.com/paper/1910.05387/) (2019)
- [Recursive Causal Structure Learning in the Presence of Latent Variables and Selection Bias](https://mlsanity.com/paper/2110.12036/) (2021)
- [NODAGS-Flow: Nonlinear Cyclic Causal Structure Learning](https://mlsanity.com/paper/2301.01849/) (2023)
- [Meta Learning for Causal Direction](https://mlsanity.com/paper/2007.02809/) (2020)
- [Causal Reasoning in the Presence of Latent Confounders via Neural ADMG Learning](https://mlsanity.com/paper/2303.12703/) (2023)
- [Federated Causal Discovery from Heterogeneous Data](https://mlsanity.com/paper/2402.13241/) (2024)
- [Causal Discovery with Multi-Domain LiNGAM for Latent Factors](https://mlsanity.com/paper/2009.09176/) (2020)
- [Independence Testing-Based Approach to Causal Discovery under Measurement Error and Linear Non-Gaussian Models](https://mlsanity.com/paper/2210.11021/) (2022)
- [Computably Continuous Reinforcement-Learning Objectives are PAC-learnable](https://mlsanity.com/paper/2303.05518/) (2023)
- [On the Reliability of Large Language Models for Causal Discovery](https://mlsanity.com/paper/2407.19638/) (2024)
- [Causal Discovery in Hawkes Processes by Minimum Description Length](https://mlsanity.com/paper/2206.06124/) (2022)

## Built on by (most cited first, 1 of 1)

- [Replicating TRACE: A Practitioner's Guide to Its Threshold and Particle Budget](https://mlsanity.com/paper/2609.01108/) (2026)

## Abstract (arXiv)

In graphical causal model, causal discovery aims to construct a causal graph based on numerical data and domain knowledge in plain text. However, the evaluation of causal discovery methods remains a challenge in the area as the progress of domain researches often makes benchmark causal graphs contain mis-aligned knowledge. This problem especially affects the evaluation of large language model (LLM) based causal discovery methods as they are sensitive to the new discoveries in the literature. This work is the first to systematically study the quality of benchmark causal graphs. Specifically, we design a pipeline that automatically retrieves relevant research papers from scientific databases, and prompts LLMs to check the consistency between the benchmark causal graphs and domain research papers. We evaluate 11 popular real-world benchmarks, for which our pipeline in total proceeds 38,081 domain papers. Our results show that popular benchmarks vary significantly in their consistency with domain research, with clear implications for causal discovery research.