mlsanityData to 14 Oct 2026

Training and Evaluating Ethical Reinforcement Learning Agents on Per-Episode Distributions

arXiv 2608.14642 ↗First posted 29 Jul 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Reinforcement Learning (RL) agents trained on a single reward signal exploit the gap between the designed reward and the intended behavior. This is particularly a problem when we are trying to imbue ethical behavior into RL agents. An agent can look ethical on average while concentrating its violations in a few bad episodes, and a creature in the environment harmed in one episode is not restored by good conduct in another. We compare four ways of training ethical behavior in Craftax, an open-ended survival benchmark. The four are: scalar penalties with termination, a linear multi-objective weight sweep, an adaptive Lagrangian constraint, and a non-compensatory utility optimized per episode under the Expected Scalarized Returns (ESR) criterion. All are evaluated under a single detector-based protocol that counts every violation in every episode without censoring. On the frontier of mean return against mean violation rate, the four methods are indistinguishable; per episode they separate sharply. At matched mean return, the ESR agent holds its stated budget of one violation in effectively every episode (worst-decile 1.04 +/- 0.07 violations), the Lagrangian leaks past the same budget (1.14 +/- 0.03), and the weight sweep's worst episodes double it (2.20 +/- 0.20). An observation-augmentation control attributes the separation to the training objective rather than to what the agent observes, and the per-episode guarantee costs nothing on the mean frontier. When ethical violations do not average away across episodes, we argue both training and evaluation must target the per-episode distribution rather than the mean.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 27 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

12 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Objective Decoupling in Social Reinforcement Learning: Recovering Ground Truth from Sycophantic Majorities8 Feb
  2. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning26 Feb

Other references · 10

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2017Safe Reinforcement Learning via Shieldingcited by 449
  4. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  5. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  6. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  7. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  8. 2020The NetHack Learning Environmentcited by 198
  9. 2019Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Surveycited by 47
  10. 2021MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learningcited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Toward Virtuous Reinforcement Learning: A Critique and Roadmap
  2. 2025Aligning Machiavellian Agents: Behavior Steering via Test-Time Policy Shaping
  3. 2025Addressing Moral Uncertainty using Large Language Models for Ethical Decision-Making
  4. 2024Reinforcement Learning and Machine ethics:a systematic review
  5. 2025Reducing Reward Dependence in RL Through Adaptive Confidence Discounting
  6. 2024Moral Alignment for LLM Agents
  7. 2026Reinforcement Learning Towards Broadly and Persistently Beneficial Models
  8. 2026Cross-Benchmark Transfer from RL on Agentic Coding Tasks