# What Does a System Modify When It Modifies Itself?

arXiv: https://arxiv.org/abs/2603.27611 · HTML: https://mlsanity.com/paper/2603.27611/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 76 references that are in the map.
- First posted on arXiv 29 Mar 2026.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [Meta-in-context learning in large language models](https://mlsanity.com/paper/2305.12907/) (2023)
- [Agent Incentives: A Causal Perspective](https://mlsanity.com/paper/2102.01685/) (2021)

## Abstract (arXiv)

When a cognitive system modifies its own functioning, what exactly does it modify: a low-level rule, a control rule, or the criterion that evaluates its revisions? Cognitive science describes executive control, metacognition, and hierarchical learning, while artificial intelligence modifies policies, parameters, and learning mechanisms, but the two fields lack common criteria for comparing these transformations. We propose a minimal analytical model distinguishing functional rules Phi_t = {R0, ..., Rk}, modification mechanisms Mt, and evaluation criteria Nt. It separates two independent dimensions: the depth of the modified target and whether the modification is blind or reflexive. Internal representational access AtR and endogenous causal control AtC are further distinguished from external inspectability and modifiability by a designer. Four regimes are defined by the target of modification: action without organizational modification, modification of low-level rules, modification of control rules or mechanisms, and revision of the evaluation criterion. Each is related to cognitive phenomena and artificial systems while making explicit the system boundary on which attribution of endogenous self-modification depends. The comparison yields a conditional crossed-opacities hypothesis: humans often have richer endogenous self-description at abstract and strategic levels than at implementation levels, whereas current artificial systems may be externally inspectable and modifiable at operational levels without corresponding self-representation or endogenous control. The framework also identifies three difficulties - viability, evaluation of revisions to evaluation criteria, and continuity of identity - and proposes two empirical discriminants for identifying the target of a modification and the causal role of self-representation.