# Mitigating loss of control in advanced AI systems through instrumental goal trajectories

arXiv: https://arxiv.org/abs/2602.01699 · HTML: https://mlsanity.com/paper/2602.01699/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 61 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Builds on (most cited first, 30 of 30)

- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)
- [Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals](https://mlsanity.com/paper/2210.01790/) (2022)
- [Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing](https://mlsanity.com/paper/2404.12253/) (2024)
- [Goal Misgeneralization in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.14111/) (2021)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [Is Power-Seeking AI an Existential Risk?](https://mlsanity.com/paper/2206.13353/) (2022)
- [Self-Improvement in Language Models: The Sharpening Mechanism](https://mlsanity.com/paper/2412.01951/) (2024)
- [Evaluating Language-Model Agents on Realistic Autonomous Tasks](https://mlsanity.com/paper/2312.11671/) (2023)
- [Fully Autonomous AI Agents Should Not be Developed](https://mlsanity.com/paper/2502.02649/) (2025)
- [The Challenge of Value Alignment: from Fairer Algorithms to AI Safety](https://mlsanity.com/paper/2101.06060/) (2021)
- [Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?](https://mlsanity.com/paper/2502.12206/) (2025)
- [The Reasons that Agents Act: Intention and Instrumental Goals](https://mlsanity.com/paper/2402.07221/) (2024)
- [Adaptive Preference Scaling for Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2406.02764/) (2024)
- [Deceptive Alignment Monitoring](https://mlsanity.com/paper/2307.10569/) (2023)
- [AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement](https://mlsanity.com/paper/2502.00757/) (2025)
- [Jailbreaking as a Reward Misspecification Problem](https://mlsanity.com/paper/2406.14393/) (2024)
- [AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations](https://mlsanity.com/paper/2406.18346/) (2024)
- [Contextual Moral Value Alignment Through Context-Based Aggregation](https://mlsanity.com/paper/2403.12805/) (2024)
- [A Review of the Evidence for Existential Risk from AI via Misaligned Power-Seeking](https://mlsanity.com/paper/2310.18244/) (2023)
- [Are Large Reasoning Models Interruptible?](https://mlsanity.com/paper/2510.11713/) (2025)
- [Self-Improvement in Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2510.02665/) (2025)

## Built on by (most cited first, 1 of 1)

- [A three-dimensional typology of agency for advanced AI systems](https://mlsanity.com/paper/2608.20041/) (2026)

## Abstract (arXiv)

Researchers at artificial intelligence labs and universities are concerned that highly capable artificial intelligence (AI) systems may erode human control by pursuing instrumental goals. Existing mitigations remain largely technical and system-centric: tracking capability in advanced systems, shaping behaviour through methods such as reinforcement learning from human feedback, and designing systems to be corrigible and interruptible. Here we develop instrumental goal trajectories to expand these options beyond the model. Gaining capability typically depends on access to additional technical resources, such as compute, storage, data and adjacent services, which in turn requires access to monetary resources. In organisations, these resources can be obtained through three organisational pathways. We label these pathways the procurement, governance and finance instrumental goal trajectories (IGTs). Each IGT produces a trail of organisational artefacts that can be monitored and used as intervention points when a systems capabilities or behaviour exceed acceptable thresholds. In this way, IGTs offer concrete avenues for defining capability levels and for broadening how corrigibility and interruptibility are implemented, shifting attention from model properties alone to the organisational systems that enable them.