# Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings

arXiv: https://arxiv.org/abs/1902.09980 · HTML: https://mlsanity.com/paper/1902.09980/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2019.

## Built on by (most cited first, 1 of 1)

- [Agent Incentives: A Causal Perspective](https://mlsanity.com/paper/2102.01685/) (2021)

## Abstract (arXiv)

Agents are systems that optimize an objective function in an environment. Together, the goal and the environment induce secondary objectives, incentives. Modeling the agent-environment interaction using causal influence diagrams, we can answer two fundamental questions about an agent's incentives directly from the graph: (1) which nodes can the agent have an incentivize to observe, and (2) which nodes can the agent have an incentivize to control? The answers tell us which information and influence points need extra protection. For example, we may want a classifier for job applications to not use the ethnicity of the candidate, and a reinforcement learning agent not to take direct control of its reward mechanism. Different algorithms and training paradigms can lead to different causal influence diagrams, so our method can be used to identify algorithms with problematic incentives and help in designing algorithms with better incentives.