# Path-Specific Objectives for Safer Agent Incentives

arXiv: https://arxiv.org/abs/2204.10018 · HTML: https://mlsanity.com/paper/2204.10018/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 21 Apr 2022.

## Builds on (most cited first, 13 of 13)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [Agent Incentives: A Causal Perspective](https://mlsanity.com/paper/2102.01685/) (2021)
- [Hidden Incentives for Auto-Induced Distributional Shift](https://mlsanity.com/paper/2009.09153/) (2020)
- [Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility](https://mlsanity.com/paper/1810.05903/) (2018)
- [Avoiding Side Effects By Considering Future Tasks](https://mlsanity.com/paper/2010.07877/) (2020)
- [Avoiding Tampering Incentives in Deep RL via Decoupled Approval](https://mlsanity.com/paper/2011.08827/) (2020)
- [Estimation of Personalized Effects Associated With Causal Pathways](https://mlsanity.com/paper/1809.10791/) (2018)

## Built on by (most cited first, 20 of 20)

- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023)
- [On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback](https://mlsanity.com/paper/2411.02306/) (2024)
- [Honesty Is the Best Policy: Defining and Mitigating AI Deception](https://mlsanity.com/paper/2312.01350/) (2023)
- [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024)
- [Counterfactual harm](https://mlsanity.com/paper/2204.12993/) (2022)
- [Discovering Agents](https://mlsanity.com/paper/2208.08345/) (2022)
- [MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking](https://mlsanity.com/paper/2501.13011/) (2025)
- [General agents contain world models](https://mlsanity.com/paper/2506.01622/) (2025)
- [Evaluating the Goal-Directedness of Large Language Models](https://mlsanity.com/paper/2504.11844/) (2025)
- [Human Control: Definitions and Algorithms](https://mlsanity.com/paper/2305.19861/) (2023)
- [Towards Measuring Goal-Directedness in AI Systems](https://mlsanity.com/paper/2410.04683/) (2024)
- [Towards Safe and Honest AI Agents with Neural Self-Other Overlap](https://mlsanity.com/paper/2412.16325/) (2024)
- [On Imperfect Recall in Multi-Agent Influence Diagrams](https://mlsanity.com/paper/2307.05059/) (2023)
- [Learning "Partner-Aware" Collaborators in Multi-Party Collaboration](https://mlsanity.com/paper/2510.22462/) (2025)
- [Solutions to preference manipulation in recommender systems require knowledge of meta-preferences](https://mlsanity.com/paper/2209.11801/) (2022)
- [Agent-Specific Effects: A Causal Effect Propagation Analysis in Multi-Agent MDPs](https://mlsanity.com/paper/2310.11334/) (2023)
- [Higher-Order Belief in Incomplete Information MAIDs](https://mlsanity.com/paper/2503.06323/) (2025)
- [Experiments with Detecting and Mitigating AI Deception](https://mlsanity.com/paper/2306.14816/) (2023)
- [Characterising Interventions in Causal Games](https://mlsanity.com/paper/2406.09318/) (2024)

## Abstract (arXiv)

We present a general framework for training safe agents whose naive incentives are unsafe. As an example, manipulative or deceptive behaviour can improve rewards but should be avoided. Most approaches fail here: agents maximize expected return by any means necessary. We formally describe settings with 'delicate' parts of the state which should not be used as a means to an end. We then train agents to maximize the causal effect of actions on the expected return which is not mediated by the delicate parts of state, using Causal Influence Diagram analysis. The resulting agents have no incentive to control the delicate state. We further show how our framework unifies and generalizes existing proposals.