# Toward Virtuous Reinforcement Learning: A Critique and Roadmap

arXiv: https://arxiv.org/abs/2512.04246 · HTML: https://mlsanity.com/paper/2512.04246/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 3 Dec 2025.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 10 of 10)

- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Emergent Social Learning via Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2010.00581/) (2020)
- [Building Ethically Bounded AI](https://mlsanity.com/paper/1812.03980/) (2018)
- [Building Ethics into Artificial Intelligence](https://mlsanity.com/paper/1812.02953/) (2018)
- [A Low-Cost Ethics Shaping Approach for Designing Reinforcement Learning Agents](https://mlsanity.com/paper/1712.04172/) (2017)
- [Reinforcement Learning Under Moral Uncertainty](https://mlsanity.com/paper/2006.04734/) (2020)
- [Joint Attention for Multi-Agent Coordination and Social Learning](https://mlsanity.com/paper/2104.07750/) (2021)
- [Towards Artificial Virtuous Agents: Games, Dilemmas and Machine Learning](https://mlsanity.com/paper/2208.14037/) (2022)
- [Reinforcement Learning and Machine ethics:a systematic review](https://mlsanity.com/paper/2407.02425/) (2024)
- [An Efficient Open World Environment for Multi-Agent Social Learning](https://mlsanity.com/paper/2508.15679/) (2025)

## Abstract (arXiv)

This paper critiques common patterns in machine ethics for Reinforcement Learning (RL) and argues for a virtue focused alternative. We highlight two recurring limitations in much of the current literature: (i) rule based (deontological) methods that encode duties as constraints or shields often struggle under ambiguity and nonstationarity and do not cultivate lasting habits, and (ii) many reward based approaches, especially single objective RL, implicitly compress diverse moral considerations into a single scalar signal, which can obscure trade offs and invite proxy gaming in practice. We instead treat ethics as policy level dispositions, that is, relatively stable habits that hold up when incentives, partners, or contexts change. This shifts evaluation beyond rule checks or scalar returns toward trait summaries, durability under interventions, and explicit reporting of moral trade offs. Our roadmap combines four components: (1) social learning in multi agent RL to acquire virtue like patterns from imperfect but normatively informed exemplars; (2) multi objective and constrained formulations that preserve value conflicts and incorporate risk aware criteria to guard against harm; (3) affinity based regularization toward updateable virtue priors that support trait like stability under distribution shift while allowing norms to evolve; and (4) operationalizing diverse ethical traditions as practical control signals, making explicit the value and cultural assumptions that shape ethical RL benchmarks.