# Measuring Goal-Directedness

arXiv: https://arxiv.org/abs/2412.04758 · HTML: https://mlsanity.com/paper/2412.04758/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Builds on (most cited first, 27 of 27)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)
- [Goal Misgeneralization in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.14111/) (2021)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [The Ethics of Advanced AI Assistants](https://mlsanity.com/paper/2404.16244/) (2024)
- [Robust agents learn causal world models](https://mlsanity.com/paper/2402.10877/) (2024)
- [Agent Incentives: A Causal Perspective](https://mlsanity.com/paper/2102.01685/) (2021)
- [Honesty Is the Best Policy: Defining and Mitigating AI Deception](https://mlsanity.com/paper/2312.01350/) (2023)
- [Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility](https://mlsanity.com/paper/1810.05903/) (2018)
- [Counterfactual harm](https://mlsanity.com/paper/2204.12993/) (2022)
- [Discovering Agents](https://mlsanity.com/paper/2208.08345/) (2022)
- [The Reasons that Agents Act: Intention and Instrumental Goals](https://mlsanity.com/paper/2402.07221/) (2024)
- [Agents and Devices: A Relative Definition of Agency](https://mlsanity.com/paper/1805.12387/) (2018)
- [A Primer on Maximum Causal Entropy Inverse Reinforcement Learning](https://mlsanity.com/paper/2203.11409/) (2022)
- [Interpreting systems as solving POMDPs: a step towards a formal understanding of agency](https://mlsanity.com/paper/2209.01619/) (2022)
- [Towards Measuring Goal-Directedness in AI Systems](https://mlsanity.com/paper/2410.04683/) (2024)
- [Interpreting Dynamical Systems as Bayesian Reasoners](https://mlsanity.com/paper/2112.13523/) (2021)
- [On Imperfect Recall in Multi-Agent Influence Diagrams](https://mlsanity.com/paper/2307.05059/) (2023)
- [Characterising Decision Theories with Mechanised Causal Graphs](https://mlsanity.com/paper/2307.10987/) (2023)
- [Formalizing Preference Utilitarianism in Physical World Models](https://mlsanity.com/paper/1504.05603/) (2015)

## Built on by (most cited first, 7 of 7)

- [Evaluating the Goal-Directedness of Large Language Models](https://mlsanity.com/paper/2504.11844/) (2025)
- [The Singapore Consensus on Global AI Safety Research Priorities](https://mlsanity.com/paper/2506.20702/) (2025)
- [Goal-Directedness is in the Eye of the Beholder](https://mlsanity.com/paper/2508.13247/) (2025)
- [A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.17697/) (2025)
- [The Impossibility of Eliciting Latent Knowledge](https://mlsanity.com/paper/2606.12268/) (2026)
- [Causal Foundations of Collective Agency](https://mlsanity.com/paper/2605.00248/) (2026)
- [A Taxonomy of Cognitive Capability Gaps in Generative and Agentic AI](https://mlsanity.com/paper/2608.02553/) (2026)

## Abstract (arXiv)

We define maximum entropy goal-directedness (MEG), a formal measure of goal-directedness in causal models and Markov decision processes, and give algorithms for computing it. Measuring goal-directedness is important, as it is a critical element of many concerns about harm from AI. It is also of philosophical interest, as goal-directedness is a key aspect of agency. MEG is based on an adaptation of the maximum causal entropy framework used in inverse reinforcement learning. It can measure goal-directedness with respect to a known utility function, a hypothesis class of utility functions, or a set of random variables. We prove that MEG satisfies several desiderata and demonstrate our algorithms with small-scale experiments.