# Self-Compatibility: Evaluating Causal Discovery without Ground Truth

arXiv: https://arxiv.org/abs/2307.09552 · HTML: https://mlsanity.com/paper/2307.09552/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 86 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 17 of 17)

- [Gradient-Based Neural DAG Learning](https://mlsanity.com/paper/1906.02226/) (2019)
- [Learning Sparse Nonparametric DAGs](https://mlsanity.com/paper/1909.13189/) (2019)
- [On the Role of Sparsity and DAG Constraints for Learning Linear DAGs](https://mlsanity.com/paper/2006.10201/) (2020)
- [Causal Inference in the Presence of Latent Variables and Selection Bias](https://mlsanity.com/paper/1302.4983/) (2013)
- [Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy To Game](https://mlsanity.com/paper/2102.13647/) (2021)
- [Foundations of Structural Causal Models with Cycles and Latent Variables](https://mlsanity.com/paper/1611.06221/) (2016)
- [Sharper bounds for uniformly stable algorithms](https://mlsanity.com/paper/1910.07833/) (2019)
- [Greedy Relaxations of the Sparsest Permutation Algorithm](https://mlsanity.com/paper/2206.05421/) (2022)
- [Nested Markov Properties for Acyclic Directed Mixed Graphs](https://mlsanity.com/paper/1701.06686/) (2017)
- [Causal Inference Through the Structural Causal Marginal Problem](https://mlsanity.com/paper/2202.01300/) (2022)
- [Telling Cause from Effect using MDL-based Local and Global Regression](https://mlsanity.com/paper/1709.08915/) (2017)
- [A Bayesian Approach to Learning Causal Networks](https://mlsanity.com/paper/1302.4958/) (2013)
- [Toward Falsifying Causal Graphs Using a Permutation-Based Test](https://mlsanity.com/paper/2305.09565/) (2023)
- [Markovian acyclic directed mixed graphs for discrete data](https://mlsanity.com/paper/1301.6624/) (2013)
- [Estimating and Controlling the False Discovery Rate for the PC Algorithm Using Edge-Specific P-Values](https://mlsanity.com/paper/1607.03975/) (2016)
- [Reinterpreting causal discovery as the task of predicting unobserved joint statistics](https://mlsanity.com/paper/2305.06894/) (2023)
- [A Robustness Test for Estimating Total Effects with Covariate Adjustment](https://mlsanity.com/paper/2206.07533/) (2022)

## Built on by (most cited first, 22 of 22)

- [CausalRivers -- Scaling up benchmarking of causal discovery for real-world time-series](https://mlsanity.com/paper/2503.17452/) (2025)
- [Sample, estimate, aggregate: A recipe for causal discovery foundation models](https://mlsanity.com/paper/2402.01929/) (2024)
- [Separation-based distance measures for causal graphs](https://mlsanity.com/paper/2402.04952/) (2024)
- [Learning Structural Causal Models through Deep Generative Models: Methods, Guarantees, and Challenges](https://mlsanity.com/paper/2405.05025/) (2024)
- [Causal Discovery over High-Dimensional Structured Hypothesis Spaces with Causal Graph Partitioning](https://mlsanity.com/paper/2406.06348/) (2024)
- [The Third Pillar of Causal Analysis? A Measurement Perspective on Causal Representations](https://mlsanity.com/paper/2505.17708/) (2025)
- [Learned Causal Method Prediction](https://mlsanity.com/paper/2311.03989/) (2023)
- [Cross-validating causal discovery via Leave-One-Variable-Out](https://mlsanity.com/paper/2411.05625/) (2024)
- [Falsification of Unconfoundedness by Testing Independence of Causal Mechanisms](https://mlsanity.com/paper/2502.06231/) (2025)
- [On Different Notions of Redundancy in Conditional-Independence-Based Discovery of Graphical Models](https://mlsanity.com/paper/2502.08531/) (2025)
- [Text Rationalization for Robust Causal Effect Estimation](https://mlsanity.com/paper/2512.05373/) (2025)
- [IV Co-Scientist: Multi-Agent LLM Framework for Causal Instrumental Variable Discovery](https://mlsanity.com/paper/2602.07943/) (2026)
- [LoSAM: Local Search in Additive Noise Models with Mixed Mechanisms and General Noise for Global Causal Discovery](https://mlsanity.com/paper/2410.11759/) (2024)
- [Causal Modeling in Multi-Context Systems: Distinguishing Multiple Context-Specific Causal Graphs which Account for Observational Support](https://mlsanity.com/paper/2410.20405/) (2024)
- [How PC-based Methods Err: Towards Better Reporting of Assumption Violations and Small Sample Errors](https://mlsanity.com/paper/2502.14719/) (2025)
- [Meta-Dependence in Conditional Independence Testing](https://mlsanity.com/paper/2504.12594/) (2025)
- [Effect-Level Validation for Causal Discovery](https://mlsanity.com/paper/2602.08340/) (2026)
- [KG-SoftMAP: Soft Knowledge-Graph Priors for Bayesian Network Structure Learning from Sparse Discrete Data](https://mlsanity.com/paper/2606.10358/) (2026)
- [Evaluating Bivariate Causal Statements Based on Mutual Compatibility](https://mlsanity.com/paper/2606.00278/) (2026)
- [A Step Towards Inherently Interpretable Causal Machine Learning Models For Decision Support](https://mlsanity.com/paper/2606.24348/) (2026)
- [Falsifying Causal Graphs With Outlier Events](https://mlsanity.com/paper/2607.12145/) (2026)
- [CAFE: Counterfactual Prediction via Fast Posterior Estimation](https://mlsanity.com/paper/2609.32167/) (2026)

## Abstract (arXiv)

As causal ground truth is incredibly rare, causal discovery algorithms are commonly only evaluated on simulated data. This is concerning, given that simulations reflect preconceptions about generating processes regarding noise distributions, model classes, and more. In this work, we propose a novel method for falsifying the output of a causal discovery algorithm in the absence of ground truth. Our key insight is that while statistical learning seeks stability across subsets of data points, causal learning should seek stability across subsets of variables. Motivated by this insight, our method relies on a notion of compatibility between causal graphs learned on different subsets of variables. We prove that detecting incompatibilities can falsify wrongly inferred causal relations due to violation of assumptions or errors from finite sample effects. Although passing such compatibility tests is only a necessary criterion for good performance, we argue that it provides strong evidence for the causal models whenever compatibility entails strong implications for the joint distribution. We also demonstrate experimentally that detection of incompatibilities can aid in causal model selection.