mlsanityData to 14 Oct 2026

Discovering Agents

arXiv 2208.08345 ↗First posted 17 Aug 2022cs.AI cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Causal models of agents have been used to analyse the safety aspects of machine learning systems. But identifying agents is non-trivial -- often the causal model is just assumed by the modeler without much justification -- and modelling failures can lead to mistakes in the safety analysis. This paper proposes the first formal causal definition of agents -- roughly that agents are systems that would adapt their policy if their actions influenced the world in a different way. From this we derive the first causal discovery algorithm for discovering agents from empirical data, and give algorithms for translating between causal models and game-theoretic influence diagrams. We demonstrate our approach by resolving some previous confusions caused by incorrect causal modelling of agents.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 21 of its 64 references that are in the map. First posted on arXiv 17 Aug 2022.

Builds on

21 of 64 references.

References · 21

  1. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  2. 2016Cooperative Inverse Reinforcement Learningcited by 300
  3. 2019Causality for Machine Learningcited by 243
  4. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  5. 2016Foundations of Structural Causal Models with Cycles and Latent Variablescited by 134
  6. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 108
  7. 2018Towards the first adversarially robust neural network model on MNISTcited by 104
  8. 2021Agent Incentives: A Causal Perspectivecited by 51
  9. 2018Constraint-based Causal Discovery for Non-Linear Structural Causal Models with Cycles and Latent Confounderscited by 35
  10. 2018Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibilitycited by 32
  11. 2021What are you optimizing for? Aligning Recommender Systems with Human Valuescited by 29
  12. 2022Estimating and Penalizing Induced Preference Shifts in Recommender Systemscited by 29
9 more
  1. 2022Counterfactual harmcited by 29
  2. 2021User Tampering in Reinforcement Learning Recommender Systemscited by 23
  3. 2022Path-Specific Objectives for Safer Agent Incentivescited by 20
  4. 2018Agents and Devices: A Relative Definition of Agencycited by 16
  5. 2021How RL Agents Behave When Their Actions Are Modifiedcited by 10
  6. 2022Why Fair Labels Can Yield Unfair Predictions: Graphical Conditions for Introduced Unfairnesscited by 10
  7. 2021Equilibrium Refinements for Multi-Agent Influence Diagrams: Theory and Practicecited by 9
  8. 2020Causal Feature Learning for Utility-Maximizing Agentscited by 5
  9. 2021Intelligence and Unambitiousness Using Algorithmic Information Theorycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 10; 2025: 8; 2026: 4.

28 papers.

Sort:

By area:

  1. 2023Model evaluation for extreme riskscited by 177
  2. 2024Robust agents learn causal world modelscited by 81
  3. 2024Visibility into AI Agentscited by 79
  4. 2025Authenticated Delegation and Authorized AI Agentscited by 65
  5. 2023Honesty Is the Best Policy: Defining and Mitigating AI Deceptioncited by 40
  6. 2026The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systemscited by 30
  7. 2026Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failurescited by 20
  8. 2025Agentic AI Needs a Systems Theorycited by 19
  9. 2024Identifiable Exchangeable Mechanisms for Causal Structure and Representation Learningcited by 16
  10. 2025Evaluating the Goal-Directedness of Large Language Modelscited by 16
  11. 2025The AI Agent Indexcited by 15
  12. 2024Three Dogmas of Reinforcement Learningcited by 14
16 more
  1. 2025HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistantscited by 11
  2. 2022Interpreting systems as solving POMDPs: a step towards a formal understanding of agencycited by 7
  3. 2024Towards Measuring Goal-Directedness in AI Systemscited by 7
  4. 2024Measuring Goal-Directednesscited by 7
  5. 2025Agency Is Frame-Dependentcited by 5
  6. 2023On Imperfect Recall in Multi-Agent Influence Diagramscited by 4
  7. 2023Characterising Decision Theories with Mechanised Causal Graphscited by 3
  8. 2025A Principle of Targeted Intervention for Multi-Agent Reinforcement Learningcited by 3
  9. 2024Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Gamescited by 1
  10. 2024Factored space models: Towards causality between levels of abstractioncited by 1
  11. 2023Experiments with Detecting and Mitigating AI Deceptioncited by 0
  12. 2024Characterising Interventions in Causal Gamescited by 0
  13. 2024Can Artificial Intelligence Embody Moral Values?cited by 0
  14. 2025Is the `Agent' Paradigm a Limiting Framework for Next-Generation Intelligent Systems?cited by 0
  15. 2026Causal Foundations of Collective Agencycited by 0
  16. 2026Artificial Id: Drive and Persistent Alignment in Agentic AIcited by 0