# Moral reinforcement learning using actual causation

arXiv: https://arxiv.org/abs/2205.08192 · HTML: https://mlsanity.com/paper/2205.08192/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 24 references that are in the map.
- First posted on arXiv 17 May 2022.

## Builds on (most cited first, 5 of 5)

- [Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning](https://mlsanity.com/paper/2010.03950/) (2020)
- [A Modification of the Halpern-Pearl Definition of Causality](https://mlsanity.com/paper/1505.00162/) (2015)
- [Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility](https://mlsanity.com/paper/1810.05903/) (2018)
- [Reinforcement Learning Under Moral Uncertainty](https://mlsanity.com/paper/2006.04734/) (2020)
- [Causality, Responsibility and Blame in Team Plans](https://mlsanity.com/paper/2005.10297/) (2020)

## Built on by (most cited first, 1 of 1)

- [Reinforcement Learning and Machine ethics:a systematic review](https://mlsanity.com/paper/2407.02425/) (2024)

## Abstract (arXiv)

Reinforcement learning systems will to a greater and greater extent make decisions that significantly impact the well-being of humans, and it is therefore essential that these systems make decisions that conform to our expectations of morally good behavior. The morally good is often defined in causal terms, as in whether one's actions have in fact caused a particular outcome, and whether the outcome could have been anticipated. We propose an online reinforcement learning method that learns a policy under the constraint that the agent should not be the cause of harm. This is accomplished by defining cause using the theory of actual causation and assigning blame to the agent when its actions are the actual cause of an undesirable outcome. We conduct experiments on a toy ethical dilemma in which a natural choice of reward function leads to clearly undesirable behavior, but our method learns a policy that avoids being the cause of harmful behavior, demonstrating the soundness of our approach. Allowing an agent to learn while observing causal moral distinctions such as blame, opens the possibility to learning policies that better conform to our moral judgments.