# Avoiding Side Effects in Complex Environments

arXiv: https://arxiv.org/abs/2006.06547 · HTML: https://mlsanity.com/paper/2006.06547/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2020.

## Built on by (most cited first, 24 of 24)

- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [B-Pref: Benchmarking Preference-Based Reinforcement Learning](https://mlsanity.com/paper/2111.03026/) (2021)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)
- [Forecasting Future World Events with Neural Networks](https://mlsanity.com/paper/2206.15474/) (2022)
- [What Would Jiminy Cricket Do? Towards Agents That Behave Morally](https://mlsanity.com/paper/2110.13136/) (2021)
- [Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking](https://mlsanity.com/paper/2403.03185/) (2024)
- [Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback](https://mlsanity.com/paper/2108.05382/) (2021)
- [Learning and Adapting Agile Locomotion Skills by Transferring Experience](https://mlsanity.com/paper/2304.09834/) (2023)
- [Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans](https://mlsanity.com/paper/2209.13020/) (2022)
- [On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Tasks](https://mlsanity.com/paper/2401.14811/) (2024)
- [SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulation](https://mlsanity.com/paper/2504.15561/) (2025)
- [Human Implicit Preference-Based Policy Fine-tuning for Multi-Agent Reinforcement Learning in USV Swarm](https://mlsanity.com/paper/2503.03796/) (2025)
- [Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification](https://mlsanity.com/paper/2403.06854/) (2024)
- [Formalizing the Problem of Side Effect Regularization](https://mlsanity.com/paper/2206.11812/) (2022)
- [Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences](https://mlsanity.com/paper/2409.07268/) (2024)
- [AI Agents Should be Regulated Based on the Extent of Their Autonomous Operations](https://mlsanity.com/paper/2503.04750/) (2025)
- [Be Considerate: Objectives, Side Effects, and Deciding How to Act](https://mlsanity.com/paper/2106.02617/) (2021)
- [STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularization](https://mlsanity.com/paper/2307.09692/) (2023)
- [Challenges for Using Impact Regularizers to Avoid Negative Side Effects](https://mlsanity.com/paper/2101.12509/) (2021)
- [Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints](https://mlsanity.com/paper/2405.03005/) (2024)
- [The Benefits of Power Regularization in Cooperative Reinforcement Learning](https://mlsanity.com/paper/2406.11240/) (2024)
- [Calibrating Conservatism for Scalable Oversight](https://mlsanity.com/paper/2605.28807/) (2026)
- [Beyond Component Testing: Validating Agentic AI Systems](https://mlsanity.com/paper/2607.29405/) (2026)

## Abstract (arXiv)

Reward function specification can be difficult. Rewarding the agent for making a widget may be easy, but penalizing the multitude of possible negative side effects is hard. In toy environments, Attainable Utility Preservation (AUP) avoided side effects by penalizing shifts in the ability to achieve randomly generated goals. We scale this approach to large, randomly generated environments based on Conway's Game of Life. By preserving optimal value for a single randomly generated reward function, AUP incurs modest overhead while leading the agent to complete the specified task and avoid many side effects. Videos and code are available at https://avoiding-side-effects.github.io/.