# Challenges for Using Impact Regularizers to Avoid Negative Side Effects

arXiv: https://arxiv.org/abs/2101.12509 · HTML: https://mlsanity.com/paper/2101.12509/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 29 Jan 2021.

## Builds on (most cited first, 15 of 15)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Safety Verification of Deep Neural Networks](https://mlsanity.com/paper/1610.06940/) (2016)
- [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://mlsanity.com/paper/1707.05173/) (2017)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning](https://mlsanity.com/paper/1711.06782/) (2017)
- [Penalizing side effects using stepwise relative reachability](https://mlsanity.com/paper/1806.01186/) (2018)
- [Conservative Agency via Attainable Utility Preservation](https://mlsanity.com/paper/1902.09725/) (2019)
- [Preferences Implicit in the State of the World](https://mlsanity.com/paper/1902.04198/) (2019)
- [Avoiding Side Effects By Considering Future Tasks](https://mlsanity.com/paper/2010.07877/) (2020)
- [Avoiding Side Effects in Complex Environments](https://mlsanity.com/paper/2006.06547/) (2020)
- [Low Impact Artificial Intelligences](https://mlsanity.com/paper/1705.10720/) (2017)
- [Avoiding Negative Side Effects due to Incomplete Knowledge of AI Systems](https://mlsanity.com/paper/2008.12146/) (2020)

## Abstract (arXiv)

Designing reward functions for reinforcement learning is difficult: besides specifying which behavior is rewarded for a task, the reward also has to discourage undesired outcomes. Misspecified reward functions can lead to unintended negative side effects, and overall unsafe behavior. To overcome this problem, recent work proposed to augment the specified reward function with an impact regularizer that discourages behavior that has a big impact on the environment. Although initial results with impact regularizers seem promising in mitigating some types of side effects, important challenges remain. In this paper, we examine the main current challenges of impact regularizers and relate them to fundamental design decisions. We discuss in detail which challenges recent approaches address and which remain unsolved. Finally, we explore promising directions to overcome the unsolved challenges in preventing negative side effects with impact regularizers.