What Does a System Modify When It Modifies Itself?
Not on the map. Why?
Abstract (arXiv)
When a cognitive system modifies its own functioning, what exactly does it modify: a low-level rule, a control rule, or the criterion that evaluates its revisions? Cognitive science describes executive control, metacognition, and hierarchical learning, while artificial intelligence modifies policies, parameters, and learning mechanisms, but the two fields lack common criteria for comparing these transformations. We propose a minimal analytical model distinguishing functional rules Phi_t = {R0, ..., Rk}, modification mechanisms Mt, and evaluation criteria Nt. It separates two independent dimensions: the depth of the modified target and whether the modification is blind or reflexive. Internal representational access AtR and endogenous causal control AtC are further distinguished from external inspectability and modifiability by a designer. Four regimes are defined by the target of modification: action without organizational modification, modification of low-level rules, modification of control rules or mechanisms, and revision of the evaluation criterion. Each is related to cognitive phenomena and artificial systems while making explicit the system boundary on which attribution of endogenous self-modification depends. The comparison yields a conditional crossed-opacities hypothesis: humans often have richer endogenous self-description at abstract and strategic levels than at implementation levels, whereas current artificial systems may be externally inspectable and modifiable at operational levels without corresponding self-representation or endogenous control. The framework also identifies three difficulties - viability, evaluation of revisions to evaluation criteria, and continuity of identity - and proposes two empirical discriminants for identifying the target of a modification and the causal role of self-representation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 76 references that are in the map. First posted on arXiv 29 Mar 2026.
Builds on
10 of 76 references.
References · 10
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2016Concrete Problems in AI Safetycited by 1,642
- 2018World Modelscited by 1,513
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
- 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
- 2016Learning to reinforcement learncited by 377
- 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
- 2023Meta-in-context learning in large language modelscited by 59
- 2021Agent Incentives: A Causal Perspectivecited by 51
Built on by
Not cited yet.