mlsanityData to 7 Oct 2026

Agent Incentives: A Causal Perspective

arXiv 2102.01685 ↗First posted 2 Feb 2021cs.AI cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

We present a framework for analysing agent incentives using causal influence diagrams. We establish that a well-known criterion for value of information is complete. We propose a new graphical criterion for value of control, establishing its soundness and completeness. We also introduce two new concepts for incentive analysis: response incentives indicate which changes in the environment affect an optimal decision, while instrumental control incentives establish whether an agent can influence its utility via a variable X. For both new concepts, we provide sound and complete graphical criteria. We show by example how these results can help with evaluating the safety and fairness of an AI system.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 18 of its 45 references that are in the map. First posted on arXiv 2 Feb 2021.

Builds on

18 of 45 references.

References · 18

  1. 2017Counterfactual Fairnesscited by 802
  2. 2013Direct and Indirect Effectscited by 276
  3. 2017Avoiding Discrimination through Causal Reasoningcited by 191
  4. 2017Fair Inference On Outcomescited by 143
  5. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  6. 2016The Off-Switch Gamecited by 97
  7. 2016A causal framework for discovering and removing direct and indirect discriminationcited by 71
  8. 2013Causal Networks: Semantics and Expressivenesscited by 67
  9. 2013Causal Discovery from Changescited by 52
  10. 2018Learning Optimal Fair Policiescited by 49
  11. 2016The AGI Containment Problemcited by 36
  12. 2013Bayes-Ball: The Rational Pastime (for Determining Irrelevance and Requisite Information in Belief Networks and Influence Diagrams)cited by 18
6 more
  1. 2019Modeling AGI Safety Frameworks with Causal Influence Diagramscited by 8
  2. 2020AGI Agent Safety by Iteratively Improving the Utility Functioncited by 8
  3. 2019Asymptotically Unambitious Artificial General Intelligencecited by 7
  4. 2013A Decision-Based View of Causalitycited by 3
  5. 2013Welldefined Decision Scenarioscited by 2
  6. 2019Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settingscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 12; 2024: 8; 2025: 11; 2026: 8.

51 papers.

Sort:

By area:

  1. 2021Alignment of Language Agentscited by 167
  2. 2023Model evaluation for extreme riskscited by 167
  3. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  4. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  5. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 94
  6. 2024Robust agents learn causal world modelscited by 81
  7. 2025Why human-AI relationships need socioaffective alignmentcited by 47
  8. 2024Beyond Preferences in AI Alignmentcited by 46
  9. 2023Honesty Is the Best Policy: Defining and Mitigating AI Deceptioncited by 40
  10. 2022Discovering Agentscited by 28
  11. 2021User Tampering in Reinforcement Learning Recommender Systemscited by 23
  12. 2022Path-Specific Objectives for Safer Agent Incentivescited by 20
38 more
  1. 2023Reasoning about Causality in Gamescited by 20
  2. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
  3. 2026Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failurescited by 18
  4. 2023Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Modelcited by 14
  5. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 12
  6. 2022Why Fair Labels Can Yield Unfair Predictions: Graphical Conditions for Introduced Unfairnesscited by 10
  7. 2023Human Control: Definitions and Algorithmscited by 10
  8. 2021Equilibrium Refinements for Multi-Agent Influence Diagrams: Theory and Practicecited by 9
  9. 2023Designing Fiduciary Artificial Intelligencecited by 8
  10. 2024Measuring Goal-Directednesscited by 7
  11. 2022A Complete Criterion for Value of Information in Soluble Influence Diagramscited by 5
  12. 2025Enhancing LLM Agent Safety via Causal Influence Promptingcited by 5
  13. 2025Agent Identity Evals: Measuring Agentic Identitycited by 5
  14. 2023On Imperfect Recall in Multi-Agent Influence Diagramscited by 4
  15. 2021Intelligence and Unambitiousness Using Algorithmic Information Theorycited by 3
  16. 2022Scoring Rules for Performative Binary Predictioncited by 3
  17. 2023Characterising Decision Theories with Mechanised Causal Graphscited by 3
  18. 2024Possible Principles for Aligned Structure Learning Agentscited by 3
  19. 2025Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Riskscited by 3
  20. 2025The Limits of Predicting Agents from Behaviourcited by 3
  21. 2025Goal-Directedness is in the Eye of the Beholdercited by 3
  22. 2022Solutions to preference manipulation in recommender systems require knowledge of meta-preferencescited by 1
  23. 2023State Representations as Incentives for Reinforcement Learning Agents: A Sim2Real Analysis on Robotic Graspingcited by 1
  24. 2023Agent-Specific Effects: A Causal Effect Propagation Analysis in Multi-Agent MDPscited by 1
  25. 2024Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Gamescited by 1
  26. 2024Factored space models: Towards causality between levels of abstractioncited by 1
  27. 2025Higher-Order Belief in Incomplete Information MAIDscited by 1
  28. 2025On Transportability for Structural Causal Banditscited by 1
  29. 2026The Impossibility of Eliciting Latent Knowledgecited by 1
  30. 2022Active learning of causal probability treescited by 0
  31. 2023Unfair Utilities and First Steps Towards Improving Themcited by 0
  32. 2024Characterising Interventions in Causal Gamescited by 0
  33. 2025MuPlon: Multi-Path Causal Optimization for Claim Verification through Controlling Confoundingcited by 0
  34. 2026Planning under Distribution Shifts with Causal POMDPscited by 0
  35. 2026What Does a System Modify When It Modifies Itself?cited by 0
  36. 2026Causal Foundations of Collective Agencycited by 0
  37. 2026The Design and Composition of Structural Causal Decision Processescited by 0
  38. 2026A Causal Markov Condition for Valuecited by 0

50 most cited of 51 All 51 (JSON)