mlsanityData to 14 Oct 2026

Penalizing side effects using stepwise relative reachability

arXiv 1806.01186 ↗First posted 4 Jun 2018cs.LG cs.AI stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

How can we design safe reinforcement learning agents that avoid unnecessary disruptions to their environment? We show that current approaches to penalizing side effects can introduce bad incentives, e.g. to prevent any irreversible changes in the environment, including the actions of other agents. To isolate the source of such undesirable incentives, we break down side effects penalties into two components: a baseline state and a measure of deviation from this baseline state. We argue that some of these incentives arise from the choice of baseline, and others arise from the choice of deviation measure. We introduce a new variant of the stepwise inaction baseline and a new deviation measure based on relative reachability of states. The combination of these design choices avoids the given undesirable incentives, while simpler baselines and the unreachability measure fail. We demonstrate this empirically by comparing different combinations of baseline and deviation measure choices on a set of gridworld experiments designed to illustrate possible bad incentives.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 6; 2022: 4; 2023: 5; 2024: 9; 2025: 7; 2026: 6.

38 papers.

Sort:

By area:

  1. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  2. 2022Defining and Characterizing Reward Hackingcited by 203
  3. 2021Consequences of Misaligned AIcited by 80
  4. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 80
  5. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  6. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  7. 2022Estimating and Penalizing Induced Preference Shifts in Recommender Systemscited by 29
  8. 2022When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learningcited by 21
  9. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Taskscited by 9
  10. 2023Designing Fiduciary Artificial Intelligencecited by 8
  11. 2021Combining Reward Information from Multiple Sourcescited by 7
  12. 2025Core Safety Values for Provably Corrigible Agentscited by 5
26 more
  1. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecificationcited by 4
  2. 2024Partial Identifiability and Misspecification in Inverse Reinforcement Learningcited by 4
  3. 2022Formalizing the Problem of Side Effect Regularizationcited by 3
  4. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
  5. 2024Possible Principles for Aligned Structure Learning Agentscited by 3
  6. 2025Safety Must Precede the Deployment of Open-Ended AIcited by 3
  7. 2021Counterfactual Planning in AGI Systemscited by 2
  8. 2025AI Agents Should be Regulated Based on the Extent of Their Autonomous Operationscited by 2
  9. 2026Robust Optimization for Mitigating Reward Hacking with Correlated Proxiescited by 2
  10. 2021Be Considerate: Objectives, Side Effects, and Deciding How to Actcited by 1
  11. 2025NP-Hard Lower Bound Complexity for Semantic Self-Verificationcited by 1
  12. 2025When Assisting One Disempowers Anothercited by 1
  13. 2021Challenges for Using Impact Regularizers to Avoid Negative Side Effectscited by 0
  14. 2021Is the Rush to Machine Learning Jeopardizing Safety? Results of a Surveycited by 0
  15. 2023Low impact agency: review and discussioncited by 0
  16. 2023Safe MDP Planning by Learning Temporal Patterns of Undesirable Trajectories and Averting Negative Side Effectscited by 0
  17. 2023Environmental-Impact Based Multi-Agent Reinforcement Learningcited by 0
  18. 2023Risk-averse Batch Active Inverse Reward Designcited by 0
  19. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraintscited by 0
  20. 2025TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labelscited by 0
  21. 2025Cognitive Silicon: An Architectural Blueprint for Post-Industrial Computing Systemscited by 0
  22. 2026AI Space Physics: Constitutive boundary semantics for open AI institutionscited by 0
  23. 2026Calibrating Conservatism for Scalable Oversightcited by 0
  24. 2026Individual Disempowerment through an Advice Channel: Control Loss when Influence is Endogenouscited by 0
  25. 2026HarvestBench: Measuring Whether LLM Agents Will Pay to Avoid Killing Animalscited by 0
  26. 2026Ajar: Measuring Open Privilege in Agent Defensescited by 0