# Self-Modification of Policy and Utility Function in Rational Agents

arXiv: https://arxiv.org/abs/1605.03142 · HTML: https://mlsanity.com/paper/1605.03142/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2016.

## Built on by (most cited first, 11 of 11)

- [Reward Shaping to Mitigate Reward Hacking in RLHF](https://mlsanity.com/paper/2502.18770/) (2025)
- [Huxley-G"odel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine](https://mlsanity.com/paper/2510.21614/) (2025)
- [The Singapore Consensus on Global AI Safety Research Priorities](https://mlsanity.com/paper/2506.20702/) (2025)
- [Towards Shutdownable Agents via Stochastic Choice](https://mlsanity.com/paper/2407.00805/) (2024)
- [A Model-Free Universal AI](https://mlsanity.com/paper/2602.23242/) (2026)
- [Deconstructing Superintelligence: Identity, Self-Modification and Diff'erance](https://mlsanity.com/paper/2604.19845/) (2026)
- [Sustaining AI safety: Control-theoretic external impossibility, intrinsic necessity, and structural requirements](https://mlsanity.com/paper/2605.12963/) (2026)
- [Modification-Considering Value Learning for Reward Hacking Mitigation in RL](https://mlsanity.com/paper/2606.28955/) (2026)
- [Recursive Criticality of AI Self-Improvement](https://mlsanity.com/paper/2609.00137/) (2026)
- [Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement](https://mlsanity.com/paper/2609.13406/) (2026)
- [Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation](https://mlsanity.com/paper/2609.31186/) (2026)

## Abstract (arXiv)

Any agent that is part of the environment it interacts with and has versatile actuators (such as arms and fingers), will in principle have the ability to self-modify -- for example by changing its own source code. As we continue to create more and more intelligent agents, chances increase that they will learn about this ability. The question is: will they want to use it? For example, highly intelligent systems may find ways to change their goals to something more easily achievable, thereby `escaping' the control of their designers. In an important paper, Omohundro (2008) argued that goal preservation is a fundamental drive of any intelligent system, since a goal is more likely to be achieved if future versions of the agent strive towards the same goal. In this paper, we formalise this argument in general reinforcement learning, and explore situations where it fails. Our conclusion is that the self-modification possibility is harmless if and only if the value function of the agent anticipates the consequences of self-modifications and use the current utility function when evaluating the future.