mlsanityData to 7 Oct 2026

Path-Specific Objectives for Safer Agent Incentives

arXiv 2204.10018 ↗First posted 21 Apr 2022cs.AI stat.MLcited by 20

Not on the map. Why?

Abstract (arXiv)

We present a general framework for training safe agents whose naive incentives are unsafe. As an example, manipulative or deceptive behaviour can improve rewards but should be avoided. Most approaches fail here: agents maximize expected return by any means necessary. We formally describe settings with 'delicate' parts of the state which should not be used as a means to an end. We then train agents to maximize the causal effect of actions on the expected return which is not mediated by the delicate parts of state, using Causal Influence Diagram analysis. The resulting agents have no incentive to control the delicate state. We further show how our framework unifies and generalizes existing proposals.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 13 of its 30 references that are in the map. First posted on arXiv 21 Apr 2022.

Builds on

13 of 30 references.

References · 13

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  4. 2017Population Based Training of Neural Networkscited by 351
  5. 2013Direct and Indirect Effectscited by 276
  6. 2021Alignment of Language Agentscited by 167
  7. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  8. 2021Agent Incentives: A Causal Perspectivecited by 51
  9. 2020Hidden Incentives for Auto-Induced Distributional Shiftcited by 42
  10. 2018Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibilitycited by 32
  11. 2020Avoiding Side Effects By Considering Future Taskscited by 32
  12. 2020Avoiding Tampering Incentives in Deep RL via Decoupled Approvalcited by 12
1 more
  1. 2018Estimation of Personalized Effects Associated With Causal Pathwayscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 6; 2024: 5; 2025: 5.

20 papers.

Sort:

By area:

  1. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  2. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  3. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
  4. 2023Honesty Is the Best Policy: Defining and Mitigating AI Deceptioncited by 40
  5. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  6. 2022Counterfactual harmcited by 29
  7. 2022Discovering Agentscited by 28
  8. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
  9. 2025Evaluating the Goal-Directedness of Large Language Modelscited by 15
  10. 2025General agents contain world modelscited by 14
  11. 2023Human Control: Definitions and Algorithmscited by 10
  12. 2024Towards Measuring Goal-Directedness in AI Systemscited by 7
8 more
  1. 2024Towards Safe and Honest AI Agents with Neural Self-Other Overlapcited by 6
  2. 2023On Imperfect Recall in Multi-Agent Influence Diagramscited by 4
  3. 2025Learning "Partner-Aware" Collaborators in Multi-Party Collaborationcited by 2
  4. 2022Solutions to preference manipulation in recommender systems require knowledge of meta-preferencescited by 1
  5. 2023Agent-Specific Effects: A Causal Effect Propagation Analysis in Multi-Agent MDPscited by 1
  6. 2025Higher-Order Belief in Incomplete Information MAIDscited by 1
  7. 2023Experiments with Detecting and Mitigating AI Deceptioncited by 0
  8. 2024Characterising Interventions in Causal Gamescited by 0