# Reinforcement Learning Under Moral Uncertainty

arXiv: https://arxiv.org/abs/2006.04734 · HTML: https://mlsanity.com/paper/2006.04734/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2020.

## Built on by (most cited first, 16 of 16)

- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans](https://mlsanity.com/paper/2209.13020/) (2022)
- [Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2303.02891/) (2023)
- [Impossibility Results in AI: A Survey](https://mlsanity.com/paper/2109.00484/) (2021)
- [Hybrid Approaches for Moral Value Alignment in AI Agents: a Manifesto](https://mlsanity.com/paper/2312.01818/) (2023)
- [Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2210.03729/) (2022)
- [Machine Love](https://mlsanity.com/paper/2302.09248/) (2023)
- [Addressing Moral Uncertainty using Large Language Models for Ethical Decision-Making](https://mlsanity.com/paper/2503.05724/) (2025)
- [Moral reinforcement learning using actual causation](https://mlsanity.com/paper/2205.08192/) (2022)
- [Advising Autonomous Cars about the Rules of the Road](https://mlsanity.com/paper/2209.14035/) (2022)
- [Moral Uncertainty and the Problem of Fanaticism](https://mlsanity.com/paper/2312.11589/) (2023)
- [Intuitions of Compromise: Utilitarianism vs. Contractualism](https://mlsanity.com/paper/2410.05496/) (2024)
- [Uncertain Machine Ethical Decisions Using Hypothetical Retrospection](https://mlsanity.com/paper/2305.01424/) (2023)
- [Uncertain Machine Ethics Planning](https://mlsanity.com/paper/2505.04352/) (2025)
- [Toward Virtuous Reinforcement Learning: A Critique and Roadmap](https://mlsanity.com/paper/2512.04246/) (2025)

## Abstract (arXiv)

An ambitious goal for machine learning is to create agents that behave ethically: The capacity to abide by human moral norms would greatly expand the context in which autonomous agents could be practically and safely deployed, e.g. fully autonomous vehicles will encounter charged moral decisions that complicate their deployment. While ethical agents could be trained by rewarding correct behavior under a specific moral theory (e.g. utilitarianism), there remains widespread disagreement about the nature of morality. Acknowledging such disagreement, recent work in moral philosophy proposes that ethical behavior requires acting under moral uncertainty, i.e. to take into account when acting that one's credence is split across several plausible ethical theories. This paper translates such insights to the field of reinforcement learning, proposes two training methods that realize different points among competing desiderata, and trains agents in simple environments to act under moral uncertainty. The results illustrate (1) how such uncertainty can help curb extreme behavior from commitment to single theories and (2) several technical complications arising from attempting to ground moral philosophy in RL (e.g. how can a principled trade-off between two competing but incomparable reward functions be reached). The aim is to catalyze progress towards morally-competent agents and highlight the potential of RL to contribute towards the computational grounding of moral philosophy.