# Effect-Level Validation for Causal Discovery

arXiv: https://arxiv.org/abs/2602.08340 · HTML: https://mlsanity.com/paper/2602.08340/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Builds on (most cited first, 9 of 9)

- [DoWhy: An End-to-End Library for Causal Inference](https://mlsanity.com/paper/2011.04216/) (2020)
- [Greedy Relaxations of the Sparsest Permutation Algorithm](https://mlsanity.com/paper/2206.05421/) (2022)
- [DoWhy-GCM: An extension of DoWhy for causal inference in graphical causal models](https://mlsanity.com/paper/2206.06821/) (2022)
- [Fast Scalable and Accurate Discovery of DAGs Using the Best Order Score Search and Grow-Shrink Trees](https://mlsanity.com/paper/2310.17679/) (2023)
- [The Case for Evaluating Causal Models Using Interventional Measures and Empirical Data](https://mlsanity.com/paper/1910.05387/) (2019)
- [Self-Compatibility: Evaluating Causal Discovery without Ground Truth](https://mlsanity.com/paper/2307.09552/) (2023)
- [CausIL: Causal Graph for Instance Level Microservice Data](https://mlsanity.com/paper/2303.00554/) (2023)
- [Efficient Latent Variable Causal Discovery: Combining Score Search and Targeted Testing](https://mlsanity.com/paper/2510.04263/) (2025)
- [Achievement and Friends: Key Factors of Player Retention Vary Across Player Levels in Online Multiplayer Games](https://mlsanity.com/paper/1702.08005/) (2017)

## Built on by (most cited first, 1 of 1)

- [Complete Causal Identification from Ancestral Graphs under Selection Bias](https://mlsanity.com/paper/2603.26301/) (2026)

## Abstract (arXiv)

Causal discovery is increasingly applied to large-scale telemetry data to estimate the effects of user-facing interventions, yet its reliability for decision-making in feedback-driven systems with strong self-selection remains unclear. In this paper, we propose an effect-centric, admissibility-first framework that treats discovered graphs as structural hypotheses and evaluates them by identifiability, stability, and falsification rather than by graph recovery accuracy alone. Empirically, we study the effect of early exposure to competitive gameplay on short-term retention using real-world game telemetry. We find that many statistically plausible discovery outputs do not admit point-identified causal queries once minimal temporal and semantic constraints are enforced, highlighting identifiability as a critical bottleneck for decision support. When identification is possible, several algorithm families converge to similar, decision-consistent effect estimates despite producing substantially different graph structures, including cases where the direct treatment-outcome edge is absent and the effect is preserved through indirect causal pathways. These converging estimates survive placebo, subsampling, and sensitivity refutation. In contrast, other methods exhibit sporadic admissibility and threshold-sensitive or attenuated effects due to endpoint ambiguity. These results suggest that graph-level metrics alone are inadequate proxies for causal reliability for a given target query. Therefore, trustworthy causal conclusions in telemetry-driven systems require prioritizing admissibility and effect-level validation over causal structural recovery alone.