# Counterfactual harm

arXiv: https://arxiv.org/abs/2204.12993 · HTML: https://mlsanity.com/paper/2204.12993/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 128 references that are in the map.
- First posted on arXiv 27 Apr 2022.

## Builds on (most cited first, 30 of 32)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)
- [Deep Structural Causal Models for Tractable Counterfactual Inference](https://mlsanity.com/paper/2006.06485/) (2020)
- [Path-Specific Counterfactual Fairness](https://mlsanity.com/paper/1802.08139/) (2018)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [Causal Autoregressive Flows](https://mlsanity.com/paper/2011.02268/) (2020)
- [Probabilities of Causation: Bounds and Identification](https://mlsanity.com/paper/1301.3898/) (2013)
- [Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search](https://mlsanity.com/paper/1811.06272/) (2018)
- [Consequences of Misaligned AI](https://mlsanity.com/paper/2102.03896/) (2021)
- [Generate Your Counterfactuals: Towards Controlled Counterfactual Generation for Text](https://mlsanity.com/paper/2012.04698/) (2020)
- [Understanding artificial intelligence ethics and safety](https://mlsanity.com/paper/1906.05684/) (2019)
- [Counterfactual Probabilities: Computational Methods, Bounds and Applications](https://mlsanity.com/paper/1302.6784/) (2013)
- [Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution](https://mlsanity.com/paper/1801.04016/) (2018)
- [Partial Counterfactual Identification from Observational and Experimental Data](https://mlsanity.com/paper/2110.05690/) (2021)
- [Causal Reasoning from Meta-reinforcement Learning](https://mlsanity.com/paper/1901.08162/) (2019)
- [A Modification of the Halpern-Pearl Definition of Causality](https://mlsanity.com/paper/1505.00162/) (2015)
- [Counterfactual Explanations in Sequential Decision Making Under Uncertainty](https://mlsanity.com/paper/2107.02776/) (2021)
- [The Sensitivity of Counterfactual Fairness to Unmeasured Confounding](https://mlsanity.com/paper/1907.01040/) (2019)
- [Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility](https://mlsanity.com/paper/1810.05903/) (2018)
- [A Structural Causal Model for MR Images of Multiple Sclerosis](https://mlsanity.com/paper/2103.03158/) (2021)
- [Path-Specific Objectives for Safer Agent Incentives](https://mlsanity.com/paper/2204.10018/) (2022)

## Built on by (most cited first, 29 of 29)

- [Robust agents learn causal world models](https://mlsanity.com/paper/2402.10877/) (2024)
- [Counterfactual Identifiability of Bijective Causal Models](https://mlsanity.com/paper/2302.02228/) (2023)
- [Discovering Agents](https://mlsanity.com/paper/2208.08345/) (2022)
- [General agents contain world models](https://mlsanity.com/paper/2506.01622/) (2025)
- [A Causal Analysis of Harm](https://mlsanity.com/paper/2210.05327/) (2022)
- [Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Model](https://mlsanity.com/paper/2306.01424/) (2023)
- [From Imitation to Introspection: Probing Self-Consciousness in Language Models](https://mlsanity.com/paper/2410.18819/) (2024)
- [Finding Counterfactually Optimal Action Sequences in Continuous State Spaces](https://mlsanity.com/paper/2306.03929/) (2023)
- [Can a Bayesian Oracle Prevent Harm from an Agent?](https://mlsanity.com/paper/2408.05284/) (2024)
- [Quantifying Harm](https://mlsanity.com/paper/2209.15111/) (2022)
- [Human Control: Definitions and Algorithms](https://mlsanity.com/paper/2305.19861/) (2023)
- [Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets](https://mlsanity.com/paper/2406.06671/) (2024)
- [Causal Fairness for Outcome Control](https://mlsanity.com/paper/2306.05066/) (2023)
- [Measuring Goal-Directedness](https://mlsanity.com/paper/2412.04758/) (2024)
- [Harm Mitigation in Recommender Systems under User Preference Dynamics](https://mlsanity.com/paper/2406.09882/) (2024)
- [Counterfactual Effect Decomposition in Multi-Agent Sequential Decision Making](https://mlsanity.com/paper/2410.12539/) (2024)
- [The Limits of Predicting Agents from Behaviour](https://mlsanity.com/paper/2506.02923/) (2025)
- [Analyzing Intentional Behavior in Autonomous Agents under Uncertainty](https://mlsanity.com/paper/2307.01532/) (2023)
- [Do No Harm: A Counterfactual Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/2405.11669/) (2024)
- [Statistical Decision Theory with Counterfactual Loss](https://mlsanity.com/paper/2505.08908/) (2025)
- [Simulating counterfactuals](https://mlsanity.com/paper/2306.15328/) (2023)
- [Agent-Specific Effects: A Causal Effect Propagation Analysis in Multi-Agent MDPs](https://mlsanity.com/paper/2310.11334/) (2023)
- [Matchings, Predictions and Counterfactual Harm in Refugee Resettlement Processes](https://mlsanity.com/paper/2407.13052/) (2024)
- [When Assisting One Disempowers Another](https://mlsanity.com/paper/2511.04177/) (2025)
- [Teleporter Theory: A General and Simple Approach for Modeling Cross-World Counterfactual Causality](https://mlsanity.com/paper/2406.11501/) (2024)
- [Learning Counterfactual Outcomes Under Rank Preservation](https://mlsanity.com/paper/2502.06398/) (2025)
- [What is a good matching of probability measures? A counterfactual lens on transport maps](https://mlsanity.com/paper/2509.16027/) (2025)
- [Masking Causality and Conditional Dependence](https://mlsanity.com/paper/2603.06984/) (2026)
- [Path-specific harm decomposition: A partial identification framework](https://mlsanity.com/paper/2609.29938/) (2026)

## Abstract (arXiv)

To act safely and ethically in the real world, agents must be able to reason about harm and avoid harmful actions. However, to date there is no statistical method for measuring harm and factoring it into algorithmic decisions. In this paper we propose the first formal definition of harm and benefit using causal models. We show that any factual definition of harm must violate basic intuitions in certain scenarios, and show that standard machine learning algorithms that cannot perform counterfactual reasoning are guaranteed to pursue harmful policies following distributional shifts. We use our definition of harm to devise a framework for harm-averse decision making using counterfactual objective functions. We demonstrate this framework on the problem of identifying optimal drug doses using a dose-response model learned from randomized control trial data. We find that the standard method of selecting doses using treatment effects results in unnecessarily harmful doses, while our counterfactual approach allows us to identify doses that are significantly less harmful without sacrificing efficacy.