# A Low-Cost Ethics Shaping Approach for Designing Reinforcement Learning Agents

arXiv: https://arxiv.org/abs/1712.04172 · HTML: https://mlsanity.com/paper/1712.04172/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Dec 2017.

## Built on by (most cited first, 18 of 18)

- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [Mind the Gap: Offline Policy Optimization for Imperfect Rewards](https://mlsanity.com/paper/2302.01667/) (2023)
- [Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2210.03729/) (2022)
- [Reinforcement Learning Guided by Provable Normative Compliance](https://mlsanity.com/paper/2203.16275/) (2022)
- [Reinforcement Learning and Machine ethics:a systematic review](https://mlsanity.com/paper/2407.02425/) (2024)
- [Addressing Moral Uncertainty using Large Language Models for Ethical Decision-Making](https://mlsanity.com/paper/2503.05724/) (2025)
- [On Assessing The Safety of Reinforcement Learning algorithms Using Formal Methods](https://mlsanity.com/paper/2111.04865/) (2021)
- [Learning Behavioral Soft Constraints from Demonstrations](https://mlsanity.com/paper/2202.10407/) (2022)
- [Formal Ethical Obligations in Reinforcement Learning Agents: Verification and Policy Updates](https://mlsanity.com/paper/2408.00147/) (2024)
- [Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignment](https://mlsanity.com/paper/2602.14844/) (2026)
- [Adaptive reinforcement learning of multi-agent ethically-aligned behaviours: the QSOM and QDSOM algorithms](https://mlsanity.com/paper/2307.00552/) (2023)
- [GOV-REK: Governed Reward Engineering Kernels for Designing Robust Multi-Agent Reinforcement Learning Systems](https://mlsanity.com/paper/2404.01131/) (2024)
- [Can Artificial Intelligence Embody Moral Values?](https://mlsanity.com/paper/2408.12250/) (2024)
- [Building Altruistic and Moral AI Agent with Brain-inspired Emotional Empathy Mechanisms](https://mlsanity.com/paper/2410.21882/) (2024)
- [ApplE: A Modular Ontology of Applied Ethics and Event Context for Ethical Decision Modeling](https://mlsanity.com/paper/2502.05110/) (2025)
- [HAVA: Hybrid Approach to Value-Alignment through Reward Weighing for Reinforcement Learning](https://mlsanity.com/paper/2505.15011/) (2025)
- [Toward Virtuous Reinforcement Learning: A Critique and Roadmap](https://mlsanity.com/paper/2512.04246/) (2025)
- [The Alignment Target Problem: Divergent Moral Judgments of Humans, AI Systems, and Their Designers](https://mlsanity.com/paper/2604.24155/) (2026)

## Abstract (arXiv)

This paper proposes a low-cost, easily realizable strategy to equip a reinforcement learning (RL) agent the capability of behaving ethically. Our model allows the designers of RL agents to solely focus on the task to achieve, without having to worry about the implementation of multiple trivial ethical patterns to follow. Based on the assumption that the majority of human behavior, regardless which goals they are achieving, is ethical, our design integrates human policy with the RL policy to achieve the target objective with less chance of violating the ethical code that human beings normally obey.