# Embedded Agency

arXiv: https://arxiv.org/abs/1902.09469 · HTML: https://mlsanity.com/paper/1902.09469/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2019.

## Built on by (most cited first, 19 of 19)

- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [Goal Misgeneralization in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.14111/) (2021)
- [Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts](https://mlsanity.com/paper/2402.15505/) (2024)
- [On Goodhart's law, with an application to value alignment](https://mlsanity.com/paper/2410.09638/) (2024)
- [Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignment](https://mlsanity.com/paper/2406.11039/) (2024)
- [Temporal Inference with Finite Factored Sets](https://mlsanity.com/paper/2109.11513/) (2021)
- [Representation Learning on a Random Lattice](https://mlsanity.com/paper/2504.20197/) (2025)
- [Induction, Popper, and machine learning](https://mlsanity.com/paper/2110.00840/) (2021)
- [Red Teaming with Mind Reading: White-Box Adversarial Policies Against RL Agents](https://mlsanity.com/paper/2209.02167/) (2022)
- [A Theory of Bounded Inductive Rationality](https://mlsanity.com/paper/2307.05068/) (2023)
- [NP-Hard Lower Bound Complexity for Semantic Self-Verification](https://mlsanity.com/paper/2501.15446/) (2025)
- [The World Is Bigger! A Computationally-Embedded Perspective on the Big World Hypothesis](https://mlsanity.com/paper/2512.23419/) (2025)
- [A game theory for foundation models shows new paths to rational cooperation through similarity inference](https://mlsanity.com/paper/2608.03958/) (2026)
- [Arguments about Highly Reliable Agent Designs as a Useful Path to Artificial Intelligence Safety](https://mlsanity.com/paper/2201.02950/) (2022)
- [The Tao of Agency: Autotelic AI, Embedded Agency and Dissolution of the Self](https://mlsanity.com/paper/2606.19924/) (2026)
- [EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures](https://mlsanity.com/paper/2606.30219/) (2026)
- [On the Computational Tractability of Robust Bandits](https://mlsanity.com/paper/2610.08740/) (2026)

## Abstract (arXiv)

Traditional models of rational action treat the agent as though it is cleanly separated from its environment, and can act on that environment from the outside. Such agents have a known functional relationship with their environment, can model their environment in every detail, and do not need to reason about themselves or their internal parts. We provide an informal survey of obstacles to formalizing good reasoning for agents embedded in their environment. Such agents must optimize an environment that is not of type "function"; they must rely on models that fit within the modeled environment; and they must reason about themselves as just another physical system, made of parts that can be modified and that can work at cross purposes.