# Evaluation Methods and Measures for Causal Learning Algorithms

arXiv: https://arxiv.org/abs/2202.02896 · HTML: https://mlsanity.com/paper/2202.02896/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 160 references that are in the map.
- First posted on arXiv 7 Feb 2022.

## Builds on (most cited first, 30 of 64)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [DAGs with NO TEARS: Continuous Optimization for Structure Learning](https://mlsanity.com/paper/1803.01422/) (2018)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [Interpretable Counterfactual Explanations Guided by Prototypes](https://mlsanity.com/paper/1907.02584/) (2019)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [A Survey on Causal Inference](https://mlsanity.com/paper/2002.02770/) (2020)
- [Explaining Classifiers with Causal Concept Effect (CaCE)](https://mlsanity.com/paper/1907.07165/) (2019)
- [CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning](https://mlsanity.com/paper/2010.04296/) (2020)
- [Causal Bandits: Learning Good Interventions via Causal Inference](https://mlsanity.com/paper/1606.03203/) (2016)
- [Inferring Networks of Substitutable and Complementary Products](https://mlsanity.com/paper/1506.08839/) (2015)
- [Size-Invariant Graph Representations for Graph Classification Extrapolations](https://mlsanity.com/paper/2103.05045/) (2021)
- [Managing Popularity Bias in Recommender Systems with Personalized Re-ranking](https://mlsanity.com/paper/1901.07555/) (2019)
- [Causal Interpretability for Machine Learning -- Problems, Methods and Evaluation](https://mlsanity.com/paper/2003.03934/) (2020)
- [Discussion on "Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects" by Hahn, Murray and Carvalho](https://mlsanity.com/paper/2108.02836/) (2021)
- [Confounding-Robust Policy Improvement](https://mlsanity.com/paper/1805.08593/) (2018)
- [CausalML: Python Package for Causal Machine Learning](https://mlsanity.com/paper/2002.11631/) (2020)
- [PC-Fairness: A Unified Framework for Measuring Causality-based Fairness](https://mlsanity.com/paper/1910.12586/) (2019)
- [Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution](https://mlsanity.com/paper/1801.04016/) (2018)
- [Learning Causal Graphs with Small Interventions](https://mlsanity.com/paper/1511.00041/) (2015)

## Built on by (most cited first, 23 of 23)

- [A Survey on Causal Discovery Methods for I.I.D. and Time Series Data](https://mlsanity.com/paper/2303.15027/) (2023)
- [Integrating Large Language Models in Causal Discovery: A Statistical Causal Approach](https://mlsanity.com/paper/2402.01454/) (2024)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Causal Discovery for Fairness](https://mlsanity.com/paper/2206.06685/) (2022)
- [Adjustment Identification Distance: A gadjid for Causal Structure Learning](https://mlsanity.com/paper/2402.08616/) (2024)
- [Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption](https://mlsanity.com/paper/2508.08883/) (2025)
- [Optimizing Data-driven Causal Discovery Using Knowledge-guided Search](https://mlsanity.com/paper/2304.05493/) (2023)
- [CausalPlayground: Addressing Data-Generation Requirements in Cutting-Edge Causality Research](https://mlsanity.com/paper/2405.13092/) (2024)
- [CausalMan: A physics-based simulator for large-scale causality](https://mlsanity.com/paper/2502.12707/) (2025)
- [How to Enhance Causal Discrimination of Utterances: A Case on Affective Reasoning](https://mlsanity.com/paper/2305.02615/) (2023)
- [SpaCE: The Spatial Confounding Environment](https://mlsanity.com/paper/2312.00710/) (2023)
- [Spatial Deconfounder: Interference-Aware Deconfounding for Spatial Causal Inference](https://mlsanity.com/paper/2510.08762/) (2025)
- [From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communications](https://mlsanity.com/paper/2505.22311/) (2025)
- [Causality-informed Anomaly Detection in Partially Observable Sensor Networks: Moving beyond Correlations](https://mlsanity.com/paper/2507.09742/) (2025)
- [Multiple Treatments Causal Effects Estimation with Task Embeddings and Balanced Representation Learning](https://mlsanity.com/paper/2511.09814/) (2025)
- [Causal inference approach to appraise long-term effects of maintenance policy on functional performance of asphalt pavements](https://mlsanity.com/paper/2405.10329/) (2024)
- [Introducing CausalBench: A Flexible Benchmark Framework for Causal Analysis and Machine Learning](https://mlsanity.com/paper/2409.08419/) (2024)
- [A Computational Theory for Efficient Mini Agent Evaluation with Causal Guarantees](https://mlsanity.com/paper/2503.21138/) (2025)
- [Causal Machine Learning in IoT-based Engineering Problems: A Tool Comparison in the Case of Household Energy Consumption](https://mlsanity.com/paper/2505.12147/) (2025)
- [Causal-driven attribution (CDA): Estimating channel influence without user-level data](https://mlsanity.com/paper/2512.21211/) (2025)
- [ORCA: An End-to-End Interactive Copilot for Optimized Root Cause Analysis](https://mlsanity.com/paper/2605.27022/) (2026)
- [Does Topic Sentiment Cause Perceived Ideology? Comparing Human and LLM Annotations in Political News Articles](https://mlsanity.com/paper/2606.06715/) (2026)
- [From Graphs to Gradients: Physics-Inspired Structural Attribution for Cyber-Physical IoT Systems and Beyond](https://mlsanity.com/paper/2607.05563/) (2026)

## Abstract (arXiv)

The convenient access to copious multi-faceted data has encouraged machine learning researchers to reconsider correlation-based learning and embrace the opportunity of causality-based learning, i.e., causal machine learning (causal learning). Recent years have therefore witnessed great effort in developing causal learning algorithms aiming to help AI achieve human-level intelligence. Due to the lack-of ground-truth data, one of the biggest challenges in current causal learning research is algorithm evaluations. This largely impedes the cross-pollination of AI and causal inference, and hinders the two fields to benefit from the advances of the other. To bridge from conventional causal inference (i.e., based on statistical methods) to causal learning with big data (i.e., the intersection of causal inference and machine learning), in this survey, we review commonly-used datasets, evaluation methods, and measures for causal learning using an evaluation pipeline similar to conventional machine learning. We focus on the two fundamental causal-inference tasks and causality-aware machine learning tasks. Limitations of current evaluation procedures are also discussed. We then examine popular causal inference tools/packages and conclude with primary challenges and opportunities for benchmarking causal learning algorithms in the era of big data. The survey seeks to bring to the forefront the urgency of developing publicly available benchmarks and consensus-building standards for causal learning evaluation with observational data. In doing so, we hope to broaden the discussions and facilitate collaboration to advance the innovation and application of causal learning.