# The Reasons that Agents Act: Intention and Instrumental Goals

arXiv: https://arxiv.org/abs/2402.07221 · HTML: https://mlsanity.com/paper/2402.07221/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 42 references that are in the map.
- First posted on arXiv 11 Feb 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Talking About Large Language Models](https://mlsanity.com/paper/2212.03551/) (2022)
- [Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals](https://mlsanity.com/paper/2210.01790/) (2022)
- [How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions](https://mlsanity.com/paper/2309.15840/) (2023)
- [Honesty Is the Best Policy: Defining and Mitigating AI Deception](https://mlsanity.com/paper/2312.01350/) (2023)
- [User Tampering in Reinforcement Learning Recommender Systems](https://mlsanity.com/paper/2109.04083/) (2021)
- [Definitions of intent suitable for algorithms](https://mlsanity.com/paper/2106.04235/) (2021)
- [A Complete Criterion for Value of Information in Soluble Influence Diagrams](https://mlsanity.com/paper/2202.11629/) (2022)
- [On Imperfect Recall in Multi-Agent Influence Diagrams](https://mlsanity.com/paper/2307.05059/) (2023)

## Built on by (most cited first, 23 of 23)

- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Robust agents learn causal world models](https://mlsanity.com/paper/2402.10877/) (2024)
- [On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback](https://mlsanity.com/paper/2411.02306/) (2024)
- [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024)
- [General agents contain world models](https://mlsanity.com/paper/2506.01622/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [From Imitation to Introspection: Probing Self-Consciousness in Language Models](https://mlsanity.com/paper/2410.18819/) (2024)
- [HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants](https://mlsanity.com/paper/2509.08494/) (2025)
- [Measuring Goal-Directedness](https://mlsanity.com/paper/2412.04758/) (2024)
- [Position: Anthropomorphic Misalignment Research Needs Stronger Evidence](https://mlsanity.com/paper/2606.07612/) (2026)
- [Possible Principles for Aligned Structure Learning Agents](https://mlsanity.com/paper/2410.00258/) (2024)
- [Giving AI Agents Access to Cryptocurrency and Smart Contracts Creates New Vectors of AI Harm](https://mlsanity.com/paper/2507.08249/) (2025)
- [Higher-Order Belief in Incomplete Information MAIDs](https://mlsanity.com/paper/2503.06323/) (2025)
- [Mitigating loss of control in advanced AI systems through instrumental goal trajectories](https://mlsanity.com/paper/2602.01699/) (2026)
- [A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents](https://mlsanity.com/paper/2602.08964/) (2026)
- [Intentionality is a Design Decision: Measuring Functional Intentionality for Accountable AI Systems](https://mlsanity.com/paper/2605.05475/) (2026)
- [The Impossibility of Eliciting Latent Knowledge](https://mlsanity.com/paper/2606.12268/) (2026)
- [Evaluating Language Model Character Traits](https://mlsanity.com/paper/2410.04272/) (2024)
- [An Aristotelian ontology of instrumental goals: Structural features to be managed and not failures to be eliminated](https://mlsanity.com/paper/2510.25471/) (2025)
- [Creativity Reconsidered: Generative AI and the Problem of Intentional Agency](https://mlsanity.com/paper/2601.15797/) (2026)
- [Teleological Inference in Structural Causal Models via Intentional Interventions](https://mlsanity.com/paper/2603.18968/) (2026)
- [Causal Foundations of Collective Agency](https://mlsanity.com/paper/2605.00248/) (2026)
- [A three-dimensional typology of agency for advanced AI systems](https://mlsanity.com/paper/2608.20041/) (2026)

## Abstract (arXiv)

Intention is an important and challenging concept in AI. It is important because it underlies many other concepts we care about, such as agency, manipulation, legal responsibility, and blame. However, ascribing intent to AI systems is contentious, and there is no universally accepted theory of intention applicable to AI agents. We operationalise the intention with which an agent acts, relating to the reasons it chooses its decision. We introduce a formal definition of intention in structural causal influence models, grounded in the philosophy literature on intent and applicable to real-world machine learning systems. Through a number of examples and results, we show that our definition captures the intuitive notion of intent and satisfies desiderata set-out by past work. In addition, we show how our definition relates to past concepts, including actual causality, and the notion of instrumental goals, which is a core idea in the literature on safe AI agents. Finally, we demonstrate how our definition can be used to infer the intentions of reinforcement learning agents and language models from their behaviour.