# Reinforcement Learning in Categorical Cybernetics

arXiv: https://arxiv.org/abs/2404.02688 · HTML: https://mlsanity.com/paper/2404.02688/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 45 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 16 of 16)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statistics](https://mlsanity.com/paper/1908.07021/) (2019)
- [Categorical Foundations of Gradient-Based Learning](https://mlsanity.com/paper/2103.01931/) (2021)
- [Compositional game theory](https://mlsanity.com/paper/1603.04641/) (2016)
- [Representable Markov Categories and Comparison of Statistical Experiments in Categorical Probability](https://mlsanity.com/paper/2010.07416/) (2020)
- [Profunctor Optics, a Categorical Update](https://mlsanity.com/paper/2001.07488/) (2020)
- [Value Iteration is Optic Composition](https://mlsanity.com/paper/2206.04547/) (2022)
- [Categorical semantics of compositional reinforcement learning](https://mlsanity.com/paper/2208.13687/) (2022)
- [Fundamental Components of Deep Learning: A category-theoretic approach](https://mlsanity.com/paper/2403.13001/) (2024)
- [Mathematical Foundations for a Compositional Account of the Bayesian Brain](https://mlsanity.com/paper/2212.12538/) (2022)
- [Autonomization of Monoidal Categories](https://mlsanity.com/paper/1411.3827/) (2014)
- [Bayesian open games](https://mlsanity.com/paper/1910.03656/) (2019)
- [The Compositional Structure of Bayesian Inference](https://mlsanity.com/paper/2305.06112/) (2023)
- [Space-time tradeoffs of lenses and optics via higher category theory](https://mlsanity.com/paper/2209.09351/) (2022)
- [Sequential decision problems, dependent types and generic solutions](https://mlsanity.com/paper/1610.07145/) (2016)
- [The game semantics of game theory](https://mlsanity.com/paper/1904.11287/) (2019)

## Built on by (most cited first, 4 of 4)

- [Reduce, Reuse, Recycle: Categories for Compositional Reinforcement Learning](https://mlsanity.com/paper/2408.13376/) (2024)
- [Disentangled Representations for Causal Cognition](https://mlsanity.com/paper/2407.00744/) (2024)
- [Agent policies from higher-order causal functions](https://mlsanity.com/paper/2512.10937/) (2025)
- [Generalised Bellman recurrence and three dualities in sequential decision-making](https://mlsanity.com/paper/2607.18077/) (2026)

## Abstract (arXiv)

We show that several major algorithms of reinforcement learning (RL) fit into the framework of categorical cybernetics, that is to say, parametrised bidirectional processes. We build on our previous work in which we show that value iteration can be represented by precomposition with a certain optic. The outline of the main construction in this paper is: (1) We extend the Bellman operators to parametrised optics that apply to action-value functions and depend on a sample. (2) We apply a representable contravariant functor, obtaining a parametrised function that applies the Bellman iteration. (3) This parametrised function becomes the backward pass of another parametrised optic that represents the model, which interacts with an environment via an agent. Thus, parametrised optics appear in two different ways in our construction, with one becoming part of the other. As we show, many of the major classes of algorithms in RL can be seen as different extremal cases of this general setup: dynamic programming, Monte Carlo methods, temporal difference learning, and deep RL. We see this as strong evidence that this approach is a natural one and believe that it will be a fruitful way to think about RL in the future.