# Conservative Agency via Attainable Utility Preservation

arXiv: https://arxiv.org/abs/1902.09725 · HTML: https://mlsanity.com/paper/1902.09725/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2019.

## Built on by (most cited first, 30 of 35)

- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AI](https://mlsanity.com/paper/2306.06924/) (2023)
- [The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists](https://mlsanity.com/paper/2403.04471/) (2024)
- [Recontextualization Mitigates Specification Gaming without Modifying the Specification](https://mlsanity.com/paper/2512.19027/) (2025)
- [Human Control: Definitions and Algorithms](https://mlsanity.com/paper/2305.19861/) (2023)
- [Societal Alignment Frameworks Can Improve LLM Alignment](https://mlsanity.com/paper/2503.00069/) (2025)
- [Combining Reward Information from Multiple Sources](https://mlsanity.com/paper/2103.12142/) (2021)
- [Core Safety Values for Provably Corrigible Agents](https://mlsanity.com/paper/2507.20964/) (2025)
- [FoMo Rewards: Can we cast foundation models as reward functions?](https://mlsanity.com/paper/2312.03881/) (2023)
- [Formalizing the Problem of Side Effect Regularization](https://mlsanity.com/paper/2206.11812/) (2022)
- [Counterfactual Planning in AGI Systems](https://mlsanity.com/paper/2102.00834/) (2021)
- [AI Agents Should be Regulated Based on the Extent of Their Autonomous Operations](https://mlsanity.com/paper/2503.04750/) (2025)
- [Be Considerate: Objectives, Side Effects, and Deciding How to Act](https://mlsanity.com/paper/2106.02617/) (2021)
- [Trajectory-Oriented Policy Optimization with Sparse Rewards](https://mlsanity.com/paper/2401.02225/) (2024)
- [Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Power](https://mlsanity.com/paper/2508.00159/) (2025)
- [When Assisting One Disempowers Another](https://mlsanity.com/paper/2511.04177/) (2025)
- [Challenges for Using Impact Regularizers to Avoid Negative Side Effects](https://mlsanity.com/paper/2101.12509/) (2021)
- [Demanding and Designing Aligned Cognitive Architectures](https://mlsanity.com/paper/2112.10190/) (2021)
- [Improving performance in multi-objective decision-making in Bottles environments with soft maximin approaches](https://mlsanity.com/paper/2208.04273/) (2022)
- [Low impact agency: review and discussion](https://mlsanity.com/paper/2303.03139/) (2023)
- [Safe MDP Planning by Learning Temporal Patterns of Undesirable Trajectories and Averting Negative Side Effects](https://mlsanity.com/paper/2304.03081/) (2023)
- [Proximal Policy Optimization Actual Combat: Manipulating Output Tokenizer Length](https://mlsanity.com/paper/2308.05585/) (2023)
- [Environmental-Impact Based Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2311.04240/) (2023)
- [Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints](https://mlsanity.com/paper/2405.03005/) (2024)
- [The Benefits of Power Regularization in Cooperative Reinforcement Learning](https://mlsanity.com/paper/2406.11240/) (2024)
- [Focused Skill Discovery: Learning to Control Specific State Variables while Minimizing Side Effects](https://mlsanity.com/paper/2510.04901/) (2025)
- [Corrigibility Transformation: Constructing Goals That Accept Updates](https://mlsanity.com/paper/2510.15395/) (2025)
- [Realist and Pluralist Conceptions of Intelligence and Their Implications on AI Research](https://mlsanity.com/paper/2511.15282/) (2025)
- [Shutdown Safety Valves for Advanced AI](https://mlsanity.com/paper/2603.07315/) (2026)

## Abstract (arXiv)

Reward functions are easy to misspecify; although designers can make corrections after observing mistakes, an agent pursuing a misspecified reward function can irreversibly change the state of its environment. If that change precludes optimization of the correctly specified reward function, then correction is futile. For example, a robotic factory assistant could break expensive equipment due to a reward misspecification; even if the designers immediately correct the reward function, the damage is done. To mitigate this risk, we introduce an approach that balances optimization of the primary reward function with preservation of the ability to optimize auxiliary reward functions. Surprisingly, even when the auxiliary reward functions are randomly generated and therefore uninformative about the correctly specified reward function, this approach induces conservative, effective behavior.