# Agent Incentives: A Causal Perspective

arXiv: https://arxiv.org/abs/2102.01685 · HTML: https://mlsanity.com/paper/2102.01685/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 2 Feb 2021.

## Builds on (most cited first, 18 of 18)

- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [Avoiding Discrimination through Causal Reasoning](https://mlsanity.com/paper/1706.02744/) (2017)
- [Fair Inference On Outcomes](https://mlsanity.com/paper/1705.10378/) (2017)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [A causal framework for discovering and removing direct and indirect discrimination](https://mlsanity.com/paper/1611.07509/) (2016)
- [Causal Networks: Semantics and Expressiveness](https://mlsanity.com/paper/1304.2379/) (2013)
- [Causal Discovery from Changes](https://mlsanity.com/paper/1301.2312/) (2013)
- [Learning Optimal Fair Policies](https://mlsanity.com/paper/1809.02244/) (2018)
- [The AGI Containment Problem](https://mlsanity.com/paper/1604.00545/) (2016)
- [Bayes-Ball: The Rational Pastime (for Determining Irrelevance and Requisite Information in Belief Networks and Influence Diagrams)](https://mlsanity.com/paper/1301.7412/) (2013)
- [Modeling AGI Safety Frameworks with Causal Influence Diagrams](https://mlsanity.com/paper/1906.08663/) (2019)
- [AGI Agent Safety by Iteratively Improving the Utility Function](https://mlsanity.com/paper/2007.05411/) (2020)
- [Asymptotically Unambitious Artificial General Intelligence](https://mlsanity.com/paper/1905.12186/) (2019)
- [A Decision-Based View of Causality](https://mlsanity.com/paper/1302.6816/) (2013)
- [Welldefined Decision Scenarios](https://mlsanity.com/paper/1301.6729/) (2013)
- [Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings](https://mlsanity.com/paper/1902.09980/) (2019)

## Built on by (most cited first, 30 of 51)

- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021)
- [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023)
- [Truthful AI: Developing and governing AI that does not lie](https://mlsanity.com/paper/2110.06674/) (2021)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Robust agents learn causal world models](https://mlsanity.com/paper/2402.10877/) (2024)
- [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024)
- [Why human-AI relationships need socioaffective alignment](https://mlsanity.com/paper/2502.02528/) (2025)
- [Honesty Is the Best Policy: Defining and Mitigating AI Deception](https://mlsanity.com/paper/2312.01350/) (2023)
- [Discovering Agents](https://mlsanity.com/paper/2208.08345/) (2022)
- [User Tampering in Reinforcement Learning Recommender Systems](https://mlsanity.com/paper/2109.04083/) (2021)
- [Path-Specific Objectives for Safer Agent Incentives](https://mlsanity.com/paper/2204.10018/) (2022)
- [Reasoning about Causality in Games](https://mlsanity.com/paper/2301.02324/) (2023)
- [Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures](https://mlsanity.com/paper/2606.08275/) (2026)
- [MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking](https://mlsanity.com/paper/2501.13011/) (2025)
- [Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Model](https://mlsanity.com/paper/2306.01424/) (2023)
- [Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspective](https://mlsanity.com/paper/2504.03255/) (2025)
- [Why Fair Labels Can Yield Unfair Predictions: Graphical Conditions for Introduced Unfairness](https://mlsanity.com/paper/2202.10816/) (2022)
- [Human Control: Definitions and Algorithms](https://mlsanity.com/paper/2305.19861/) (2023)
- [Equilibrium Refinements for Multi-Agent Influence Diagrams: Theory and Practice](https://mlsanity.com/paper/2102.05008/) (2021)
- [Designing Fiduciary Artificial Intelligence](https://mlsanity.com/paper/2308.02435/) (2023)
- [Measuring Goal-Directedness](https://mlsanity.com/paper/2412.04758/) (2024)
- [A Complete Criterion for Value of Information in Soluble Influence Diagrams](https://mlsanity.com/paper/2202.11629/) (2022)
- [Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Risks](https://mlsanity.com/paper/2505.19806/) (2025)
- [Enhancing LLM Agent Safety via Causal Influence Prompting](https://mlsanity.com/paper/2507.00979/) (2025)
- [Agent Identity Evals: Measuring Agentic Identity](https://mlsanity.com/paper/2507.17257/) (2025)
- [On Imperfect Recall in Multi-Agent Influence Diagrams](https://mlsanity.com/paper/2307.05059/) (2023)
- [Intelligence and Unambitiousness Using Algorithmic Information Theory](https://mlsanity.com/paper/2105.06268/) (2021)
- [Scoring Rules for Performative Binary Prediction](https://mlsanity.com/paper/2207.02847/) (2022)
- [Characterising Decision Theories with Mechanised Causal Graphs](https://mlsanity.com/paper/2307.10987/) (2023)

## Abstract (arXiv)

We present a framework for analysing agent incentives using causal influence diagrams. We establish that a well-known criterion for value of information is complete. We propose a new graphical criterion for value of control, establishing its soundness and completeness. We also introduce two new concepts for incentive analysis: response incentives indicate which changes in the environment affect an optimal decision, while instrumental control incentives establish whether an agent can influence its utility via a variable X. For both new concepts, we provide sound and complete graphical criteria. We show by example how these results can help with evaluating the safety and fairness of an AI system.