# Discovering Agents

arXiv: https://arxiv.org/abs/2208.08345 · HTML: https://mlsanity.com/paper/2208.08345/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 64 references that are in the map.
- First posted on arXiv 17 Aug 2022.

## Builds on (most cited first, 21 of 21)

- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms](https://mlsanity.com/paper/1901.10912/) (2019)
- [Foundations of Structural Causal Models with Cycles and Latent Variables](https://mlsanity.com/paper/1611.06221/) (2016)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [Towards the first adversarially robust neural network model on MNIST](https://mlsanity.com/paper/1805.09190/) (2018)
- [Agent Incentives: A Causal Perspective](https://mlsanity.com/paper/2102.01685/) (2021)
- [Constraint-based Causal Discovery for Non-Linear Structural Causal Models with Cycles and Latent Confounders](https://mlsanity.com/paper/1807.03024/) (2018)
- [Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility](https://mlsanity.com/paper/1810.05903/) (2018)
- [What are you optimizing for? Aligning Recommender Systems with Human Values](https://mlsanity.com/paper/2107.10939/) (2021)
- [Estimating and Penalizing Induced Preference Shifts in Recommender Systems](https://mlsanity.com/paper/2204.11966/) (2022)
- [Counterfactual harm](https://mlsanity.com/paper/2204.12993/) (2022)
- [User Tampering in Reinforcement Learning Recommender Systems](https://mlsanity.com/paper/2109.04083/) (2021)
- [Path-Specific Objectives for Safer Agent Incentives](https://mlsanity.com/paper/2204.10018/) (2022)
- [Agents and Devices: A Relative Definition of Agency](https://mlsanity.com/paper/1805.12387/) (2018)
- [How RL Agents Behave When Their Actions Are Modified](https://mlsanity.com/paper/2102.07716/) (2021)
- [Why Fair Labels Can Yield Unfair Predictions: Graphical Conditions for Introduced Unfairness](https://mlsanity.com/paper/2202.10816/) (2022)
- [Equilibrium Refinements for Multi-Agent Influence Diagrams: Theory and Practice](https://mlsanity.com/paper/2102.05008/) (2021)
- [Causal Feature Learning for Utility-Maximizing Agents](https://mlsanity.com/paper/2005.08792/) (2020)
- [Intelligence and Unambitiousness Using Algorithmic Information Theory](https://mlsanity.com/paper/2105.06268/) (2021)

## Built on by (most cited first, 28 of 28)

- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Robust agents learn causal world models](https://mlsanity.com/paper/2402.10877/) (2024)
- [Visibility into AI Agents](https://mlsanity.com/paper/2401.13138/) (2024)
- [Authenticated Delegation and Authorized AI Agents](https://mlsanity.com/paper/2501.09674/) (2025)
- [Honesty Is the Best Policy: Defining and Mitigating AI Deception](https://mlsanity.com/paper/2312.01350/) (2023)
- [The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems](https://mlsanity.com/paper/2602.17753/) (2026)
- [Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures](https://mlsanity.com/paper/2606.08275/) (2026)
- [Agentic AI Needs a Systems Theory](https://mlsanity.com/paper/2503.00237/) (2025)
- [Identifiable Exchangeable Mechanisms for Causal Structure and Representation Learning](https://mlsanity.com/paper/2406.14302/) (2024)
- [Evaluating the Goal-Directedness of Large Language Models](https://mlsanity.com/paper/2504.11844/) (2025)
- [The AI Agent Index](https://mlsanity.com/paper/2502.01635/) (2025)
- [Three Dogmas of Reinforcement Learning](https://mlsanity.com/paper/2407.10583/) (2024)
- [HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants](https://mlsanity.com/paper/2509.08494/) (2025)
- [Interpreting systems as solving POMDPs: a step towards a formal understanding of agency](https://mlsanity.com/paper/2209.01619/) (2022)
- [Towards Measuring Goal-Directedness in AI Systems](https://mlsanity.com/paper/2410.04683/) (2024)
- [Measuring Goal-Directedness](https://mlsanity.com/paper/2412.04758/) (2024)
- [Agency Is Frame-Dependent](https://mlsanity.com/paper/2502.04403/) (2025)
- [On Imperfect Recall in Multi-Agent Influence Diagrams](https://mlsanity.com/paper/2307.05059/) (2023)
- [Characterising Decision Theories with Mechanised Causal Graphs](https://mlsanity.com/paper/2307.10987/) (2023)
- [A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.17697/) (2025)
- [Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Games](https://mlsanity.com/paper/2405.16588/) (2024)
- [Factored space models: Towards causality between levels of abstraction](https://mlsanity.com/paper/2412.02579/) (2024)
- [Experiments with Detecting and Mitigating AI Deception](https://mlsanity.com/paper/2306.14816/) (2023)
- [Characterising Interventions in Causal Games](https://mlsanity.com/paper/2406.09318/) (2024)
- [Can Artificial Intelligence Embody Moral Values?](https://mlsanity.com/paper/2408.12250/) (2024)
- [Is the `Agent' Paradigm a Limiting Framework for Next-Generation Intelligent Systems?](https://mlsanity.com/paper/2509.10875/) (2025)
- [Causal Foundations of Collective Agency](https://mlsanity.com/paper/2605.00248/) (2026)
- [Artificial Id: Drive and Persistent Alignment in Agentic AI](https://mlsanity.com/paper/2609.11911/) (2026)

## Abstract (arXiv)

Causal models of agents have been used to analyse the safety aspects of machine learning systems. But identifying agents is non-trivial -- often the causal model is just assumed by the modeler without much justification -- and modelling failures can lead to mistakes in the safety analysis. This paper proposes the first formal causal definition of agents -- roughly that agents are systems that would adapt their policy if their actions influenced the world in a different way. From this we derive the first causal discovery algorithm for discovering agents from empirical data, and give algorithms for translating between causal models and game-theoretic influence diagrams. We demonstrate our approach by resolving some previous confusions caused by incorrect causal modelling of agents.