# Causal Testing: Finding Defects' Root Causes

arXiv: https://arxiv.org/abs/1809.06991 · HTML: https://mlsanity.com/paper/1809.06991/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2018.

## Built on by (most cited first, 22 of 22)

- [Causality-based Neural Network Repair](https://mlsanity.com/paper/2204.09274/) (2022)
- [Rango: Adaptive Retrieval-Augmented Proving for Automated Software Verification](https://mlsanity.com/paper/2412.14063/) (2024)
- [Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approach](https://mlsanity.com/paper/2310.06680/) (2023)
- [Unicorn: Reasoning about Configurable System Performance through the lens of Causality](https://mlsanity.com/paper/2201.08413/) (2022)
- [Causality Analysis for Evaluating the Security of Large Language Models](https://mlsanity.com/paper/2312.07876/) (2023)
- [Causality-Aided Trade-off Analysis for Machine Learning Fairness](https://mlsanity.com/paper/2305.13057/) (2023)
- [CaRE: Finding Root Causes of Configuration Issues in Highly-Configurable Robots](https://mlsanity.com/paper/2301.07690/) (2023)
- [QEDCartographer: Automating Formal Verification Using Reward-Free Reinforcement Learning](https://mlsanity.com/paper/2408.09237/) (2024)
- [Causal Reasoning in Software Quality Assurance: A Systematic Review](https://mlsanity.com/paper/2408.17183/) (2024)
- [Enabling Runtime Verification of Causal Discovery Algorithms with Automated Conditional Independence Reasoning (Extended Version)](https://mlsanity.com/paper/2309.05264/) (2023)
- [CURE: Simulation-Augmented Auto-Tuning in Robotics](https://mlsanity.com/paper/2402.05399/) (2024)
- [Reasoning-Based Software Testing](https://mlsanity.com/paper/2303.01302/) (2023)
- [Causal fault localisation in dataflow systems](https://mlsanity.com/paper/2304.11987/) (2023)
- [Detection and Evaluation of bias-inducing Features in Machine learning](https://mlsanity.com/paper/2310.12805/) (2023)
- [CAM: A Causality-based Analysis Framework for Multi-Agent Code Generation Systems](https://mlsanity.com/paper/2602.02138/) (2026)
- [Towards Counterfactual Explanation and Assertion Inference for CPS Debugging](https://mlsanity.com/paper/2604.07679/) (2026)
- [Causality-Driven Neural Network Repair: Challenges and Opportunities](https://mlsanity.com/paper/2504.17946/) (2025)
- [Your Model Is Unfair, Are You Even Aware? Inverse Relationship Between Comprehension and Trust in Explainability Visualizations of Biased ML Models](https://mlsanity.com/paper/2508.00140/) (2025)
- [SoK: a Comprehensive Causality Analysis Framework for Large Language Model Security](https://mlsanity.com/paper/2512.04841/) (2025)
- [Causal Software Engineering: A Vision and Roadmap](https://mlsanity.com/paper/2605.02454/) (2026)
- [Reasoning Beyond Prediction: From Data-Driven to Causal Software Engineering](https://mlsanity.com/paper/2606.27960/) (2026)
- [SWE-Game: Can Coding Agents Build the Games We Want?](https://mlsanity.com/paper/2609.33678/) (2026)

## Abstract (arXiv)

Understanding the root cause of a defect is critical to isolating and repairing buggy behavior. We present Causal Testing, a new method of root-cause analysis that relies on the theory of counterfactual causality to identify a set of executions that likely hold key causal information necessary to understand and repair buggy behavior. Using the Defects4J benchmark, we find that Causal Testing could be applied to 71% of real-world defects, and for 77% of those, it can help developers identify the root cause of the defect. A controlled experiment with 37 developers shows that Causal Testing improves participants' ability to identify the cause of the defect from 80% of the time with standard testing tools to 86% of the time with Causal Testing. The participants report that Causal Testing provides useful information they cannot get using tools such as JUnit. Holmes, our prototype, open-source Eclipse plugin implementation of Causal Testing, is available at http://holmes.cs.umass.edu/.