# Universal Reinforcement Learning in Coalgebras: Asynchronous Stochastic Computation via Conduction

arXiv: https://arxiv.org/abs/2508.15128 · HTML: https://mlsanity.com/paper/2508.15128/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 6 of its 65 references that are in the map.
- First posted on arXiv 20 Aug 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 6 of 6)

- [A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statistics](https://mlsanity.com/paper/1908.07021/) (2019)
- [Backprop as Functor: A compositional perspective on supervised learning](https://mlsanity.com/paper/1711.10455/) (2017)
- [Continuous MDP Homomorphisms and Homomorphic Policy Gradient](https://mlsanity.com/paper/2209.07364/) (2022)
- [Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces](https://mlsanity.com/paper/1405.6757/) (2014)
- [Universal Decision Models](https://mlsanity.com/paper/2110.15431/) (2021)
- [Universal Imitation Games](https://mlsanity.com/paper/2405.01540/) (2024)

## Built on by (most cited first, 2 of 2)

- [Working Paper: Towards a Category-theoretic Comparative Framework for Artificial General Intelligence](https://mlsanity.com/paper/2603.28906/) (2026)
- [Consciousness as a Functor](https://mlsanity.com/paper/2508.17561/) (2025)

## Abstract (arXiv)

In this paper, we introduce a categorial generalization of RL, termed universal reinforcement learning (URL), building on powerful mathematical abstractions from the study of coinduction on non-well-founded sets and universal coalgebras, topos theory, and categorial models of asynchronous parallel distributed computation. In the first half of the paper, we review the basic RL framework, illustrate the use of categories and functors in RL, showing how they lead to interesting insights. In particular, we also introduce a standard model of asynchronous distributed minimization proposed by Bertsekas and Tsitsiklis, and describe the relationship between metric coinduction and their proof of the Asynchronous Convergence Theorem. The space of algorithms for MDPs or PSRs can be modeled as a functor category, where the co-domain category forms a topos, which admits all (co)limits, possesses a subobject classifier, and has exponential objects. In the second half of the paper, we move on to universal coalgebras. Dynamical system models, such as Markov decision processes (MDPs), partially observed MDPs (POMDPs), a predictive state representation (PSRs), and linear dynamical systems (LDSs) are all special types of coalgebras. We describe a broad family of universal coalgebras, extending the dynamic system models studied previously in RL. The core problem in finding fixed points in RL to determine the exact or approximate (action) value function is generalized in URL to determining the final coalgebra asynchronously in a parallel distributed manner.