mlsanityData to 7 Oct 2026

Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy To Game

arXiv 2102.13647 ↗First posted 26 Feb 2021stat.ML cs.LG stat.MEcited by 159

Not on the map. Why?

Abstract (arXiv)

Simulated DAG models may exhibit properties that, perhaps inadvertently, render their structure identifiable and unexpectedly affect structure learning algorithms. Here, we show that marginal variance tends to increase along the causal order for generically sampled additive noise models. We introduce varsortability as a measure of the agreement between the order of increasing marginal variance and the causal order. For commonly sampled graphs and model parameters, we show that the remarkable performance of some continuous structure learning algorithms can be explained by high varsortability and matched by a simple baseline method. Yet, this performance may not transfer to real-world data where varsortability may be moderate or dependent on the choice of measurement scales. On standardized data, the same algorithms fail to identify the ground-truth DAG or its Markov equivalence class. While standardization removes the pattern in marginal variance, we show that data generating processes that incur high varsortability also leave a distinct covariance pattern that may be exploited even after standardization. Our findings challenge the significance of generic benchmarks with independently drawn parameters. The code is available at https://github.com/Scriddie/Varsortability.

Key facts. Cited by 159 papers in the map (data through 7 Oct 2026). Builds on 22 of its 57 references that are in the map. First posted on arXiv 26 Feb 2021.

Builds on

22 of 57 references.

References · 22

  1. 2018DAGs with NO TEARS: Continuous Optimization for Structure Learningcited by 745
  2. 2019DAG-GNN: DAG Structure Learning with Graph Neural Networkscited by 329
  3. 2019Causality for Machine Learningcited by 242
  4. 2019Gradient-Based Neural DAG Learningcited by 227
  5. 2020Differentiable Causal Discovery from Interventional Datacited by 218
  6. 2019Learning Sparse Nonparametric DAGscited by 207
  7. 2020DYNOTEARS: Structure Learning from Time-Series Datacited by 198
  8. 2020On the Role of Sparsity and DAG Constraints for Learning Linear DAGscited by 186
  9. 2013CAM: Causal additive models, high-dimensional order search and penalized regressioncited by 175
  10. 2013High-dimensional learning of linear causal networks via inverse covariance estimationcited by 86
  11. 2013Learning Equivalence Classes of Bayesian Networks Structurescited by 75
  12. 2020DAGs with No Fears: A Closer Look at Continuous Optimization for Learning Bayesian Networkscited by 66
10 more
  1. 2020Differentiable Causal Discovery Under Unmeasured Confoundingcited by 56
  2. 2018On Causal Discovery with Equal Variance Assumptioncited by 55
  3. 2017Learning linear structural equation models in polynomial time and sample complexitycited by 49
  4. 2013Strong Completeness and Faithfulness in Bayesian Networkscited by 44
  5. 2013A Transformational Characterization of Equivalent Bayesian Network Structurescited by 41
  6. 2019Scaling structural learning with NO-BEARS to infer causal transcriptome networkscited by 39
  7. 2020A polynomial-time algorithm for learning nonparametric causal graphscited by 29
  8. 2014Concave Penalized Estimation of Sparse Gaussian Bayesian Networkscited by 27
  9. 2017Learning Identifiable Gaussian Bayesian Networks in Polynomial Time and Sample Complexitycited by 27
  10. 2020Causal structure learning from time series: Large regression coefficients may predict causal links better in practice than small p-valuescited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 20; 2023: 30; 2024: 48; 2025: 31; 2026: 27.

159 papers.

Sort:

By area:

  1. 2023Learning Linear Causal Representations from Interventions under General Nonlinear Mixingcited by 92
  2. 2022On the Identifiability and Estimation of Causal Location-Scale Noise Modelscited by 56
  3. 2022Large-Scale Differentiable Causal Discovery of Factor Graphscited by 47
  4. 2023Causal Discovery from Temporal Data: An Overview and New Perspectivescited by 43
  5. 2023CausalTime: Realistically Generated Time-series for Benchmarking of Causal Discoverycited by 35
  6. 2023Structure Learning with Continuous Optimization: A Sober Look and Beyondcited by 34
  7. 2021NTS-NOTEARS: Learning Nonparametric DBNs With Prior Knowledgecited by 33
  8. 2023Discovering Causal Relations and Equations from Datacited by 33
  9. 2024Integrating Large Language Models in Causal Discovery: A Statistical Causal Approachcited by 31
  10. 2022Active Learning for Optimal Intervention Design in Causal Modelscited by 30
  11. 2023Causal Representation Learning Made Identifiable by Grouping of Observational Variablescited by 30
  12. 2023Stable Differentiable Causal Discoverycited by 30
38 more
  1. 2022CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Datacited by 29
  2. 2024ALCM: Autonomous LLM-Augmented Causal Discovery Frameworkcited by 29
  3. 2023A Scale-Invariant Sorting Criterion to Find a Causal Order in Additive Noise Modelscited by 28
  4. 2023Assumption violations in causal discovery and the robustness of score matchingcited by 28
  5. 2022BaCaDI: Bayesian Causal Discovery with Unknown Interventionscited by 26
  6. 2025CausalRivers -- Scaling up benchmarking of causal discovery for real-world time-seriescited by 24
  7. 2023Self-Compatibility: Evaluating Causal Discovery without Ground Truthcited by 22
  8. 2024A Meta-Learning Approach to Bayesian Causal Discoverycited by 22
  9. 2022MissDAG: Causal Discovery in the Presence of Missing Data with Continuous Additive Noise Modelscited by 21
  10. 2024Sample, estimate, aggregate: A recipe for causal discovery foundation modelscited by 19
  11. 2023textttcausalAssembly: Generating Realistic Production Data for Benchmarking Causal Discoverycited by 18
  12. 2024Signature Kernel Conditional Independence Tests in Causal Discovery for Stochastic Processescited by 18
  13. 2023Optimizing NOTEARS Objectives via Topological Swapscited by 17
  14. 2024Standardizing Structural Causal Modelscited by 17
  15. 2024The Landscape of Causal Discovery Data: Grounding Causal Discovery in Real-World Applicationscited by 17
  16. 2024Adjustment Identification Distance: A gadjid for Causal Structure Learningcited by 16
  17. 2024Federated Causal Discovery from Heterogeneous Datacited by 15
  18. 2022Characterization and Greedy Learning of Gaussian Structural Causal Models under Unknown Interventionscited by 12
  19. 2023CoLiDE: Concomitant Linear DAG Estimationcited by 12
  20. 2021Benchpress: A Scalable and Versatile Workflow for Benchmarking Structure Learning Algorithmscited by 11
  21. 2022A Meta-Reinforcement Learning Algorithm for Causal Discoverycited by 11
  22. 2022GFlowCausal: Generative Flow Networks for Causal Discoverycited by 11
  23. 2023Bivariate Causal Discovery using Bayesian Model Selectioncited by 11
  24. 2024Ordering-Based Causal Discovery for Linear and Nonlinear Relationscited by 11
  25. 2023Constraint-Free Structure Learning with Smooth Acyclic Orientationscited by 10
  26. 2024Amortized Active Causal Induction with Deep Reinforcement Learningcited by 10
  27. 2024Demystifying amortized causal discovery with transformerscited by 10
  28. 2024Identifying Drift, Diffusion, and Causal Structure from Temporal Snapshotscited by 10
  29. 2022Identifying Patient-Specific Root Causes of Diseasecited by 8
  30. 2023Evaluation of Induced Expert Knowledge in Causal Structure Learning by NOTEARScited by 8
  31. 2023Learning DAGs from Data with Few Root Causescited by 8
  32. 2023Effective Causal Discovery under Identifiable Heteroscedastic Noise Modelcited by 8
  33. 2024Learning Structural Causal Models through Deep Generative Models: Methods, Guarantees, and Challengescited by 8
  34. 2024DCILP: A Distributed Approach for Large-Scale Causal Structure Learningcited by 8
  35. 2024Score matching through the roof: linear, nonlinear, and latent variables causal discoverycited by 8
  36. 2023CDANs: Temporal Causal Discovery from Autocorrelated and Non-Stationary Time Series Datacited by 7
  37. 2023Open problems in causal structure learning: A case study of COVID-19 in the UKcited by 7
  38. 2023RCT Rejection Sampling for Causal Estimation Evaluationcited by 7

50 most cited of 159 All 159 (JSON)