mlsanityData to 7 Oct 2026

Avoiding Side Effects in Complex Environments

arXiv 2006.06547 ↗First posted 11 Jun 2020cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Reward function specification can be difficult. Rewarding the agent for making a widget may be easy, but penalizing the multitude of possible negative side effects is hard. In toy environments, Attainable Utility Preservation (AUP) avoided side effects by penalizing shifts in the ability to achieve randomly generated goals. We scale this approach to large, randomly generated environments based on Conway's Game of Life. By preserving optimal value for a single randomly generated reward function, AUP incurs modest overhead while leading the agent to complete the specified task and avoid many side effects. Videos and code are available at https://avoiding-side-effects.github.io/.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 4; 2023: 2; 2024: 6; 2025: 3; 2026: 2.

24 papers.

Sort:

By area:

  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  2. 2021Unsolved Problems in ML Safetycited by 291
  3. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  4. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  5. 2022Forecasting Future World Events with Neural Networkscited by 63
  6. 2021What Would Jiminy Cricket Do? Towards Agents That Behave Morallycited by 57
  7. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  8. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedbackcited by 34
  9. 2023Learning and Adapting Agile Locomotion Skills by Transferring Experiencecited by 25
  10. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  11. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Taskscited by 9
  12. 2025SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulationcited by 9
12 more
  1. 2025Human Implicit Preference-Based Policy Fine-tuning for Multi-Agent Reinforcement Learning in USV Swarmcited by 5
  2. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecificationcited by 4
  3. 2022Formalizing the Problem of Side Effect Regularizationcited by 3
  4. 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferencescited by 2
  5. 2025AI Agents Should be Regulated Based on the Extent of Their Autonomous Operationscited by 2
  6. 2021Be Considerate: Objectives, Side Effects, and Deciding How to Actcited by 1
  7. 2023STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularizationcited by 1
  8. 2021Challenges for Using Impact Regularizers to Avoid Negative Side Effectscited by 0
  9. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraintscited by 0
  10. 2024The Benefits of Power Regularization in Cooperative Reinforcement Learningcited by 0
  11. 2026Calibrating Conservatism for Scalable Oversightcited by 0
  12. 2026Beyond Component Testing: Validating Agentic AI Systemscited by 0