# Agent policies from higher-order causal functions

arXiv: https://arxiv.org/abs/2512.10937 · HTML: https://mlsanity.com/paper/2512.10937/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 44 references that are in the map.
- First posted on arXiv 11 Dec 2025.

## Builds on (most cited first, 15 of 15)

- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent States](https://mlsanity.com/paper/2102.05261/) (2021)
- [Agent-state based policies in POMDPs: Beyond belief-state MDPs](https://mlsanity.com/paper/2409.15703/) (2024)
- [An Introduction to Quantum Reinforcement Learning (QRL)](https://mlsanity.com/paper/2409.05846/) (2024)
- [Periodic agent-state based Q-learning for POMDPs](https://mlsanity.com/paper/2407.06121/) (2024)
- [Value Iteration is Optic Composition](https://mlsanity.com/paper/2206.04547/) (2022)
- [Reinforcement Learning in Categorical Cybernetics](https://mlsanity.com/paper/2404.02688/) (2024)
- [A categorical semantics for causal structure](https://mlsanity.com/paper/1701.04732/) (2017)
- [Quantum Agents for Algorithmic Discovery](https://mlsanity.com/paper/2510.08159/) (2025)
- [Equivalence of Grandfather and Information Antinomy Under Intervention](https://mlsanity.com/paper/2004.12921/) (2020)
- [Higher-order causal theories are models of BV-logic](https://mlsanity.com/paper/2205.11219/) (2022)
- [A Profunctorial Semantics for Quantum Supermaps](https://mlsanity.com/paper/2402.02997/) (2024)
- [A complete logic for causal consistency](https://mlsanity.com/paper/2403.09297/) (2024)
- [Quantum advantage in decentralized control of POMDPs: A control-theoretic view of the Mermin-Peres square](https://mlsanity.com/paper/2501.16690/) (2025)
- [A BV-Category of Spacetime Interventions](https://mlsanity.com/paper/2502.19022/) (2025)

## Abstract (arXiv)

We establish a correspondence between equivalence classes of agent-state policies for deterministic POMDPs and one-input process functions (the classical-deterministic limit of higher-order quantum operations). We use this correspondence to build a bridge between the agent-environment interaction in artificial intelligence, causal structure in the foundations of physics, and logic in computer science. We construct a *-autonomous category PF of types which supports an interpretation of one-step evaluation of policies, and multi-agent observation constraints, into cuts and monoidal products. In terms of types, we develop the correspondence further by identifying observation-independent decentralised POMDPs as the natural domain for the multi-input process functions used to model indefinite causality. We then prove a strict separation between general multi-input process function and definite-ordered process function performance on such dec-POMDPs, by finding an instance for which policies utilizing an indefinite causal structure can achieve greater finite-horizon rewards than policies which are restricted to a fixed background causal structure.