# Value Iteration is Optic Composition

arXiv: https://arxiv.org/abs/2206.04547 · HTML: https://mlsanity.com/paper/2206.04547/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2022.

## Built on by (most cited first, 5 of 5)

- [Reinforcement Learning in Categorical Cybernetics](https://mlsanity.com/paper/2404.02688/) (2024)
- [The Compositional Structure of Bayesian Inference](https://mlsanity.com/paper/2305.06112/) (2023)
- [Space-time tradeoffs of lenses and optics via higher category theory](https://mlsanity.com/paper/2209.09351/) (2022)
- [Agent policies from higher-order causal functions](https://mlsanity.com/paper/2512.10937/) (2025)
- [Generalised Bellman recurrence and three dualities in sequential decision-making](https://mlsanity.com/paper/2607.18077/) (2026)

## Abstract (arXiv)

Dynamic programming is a class of algorithms used to compute optimal control policies for Markov decision processes. Dynamic programming is ubiquitous in control theory, and is also the foundation of reinforcement learning. In this paper, we show that value improvement, one of the main steps of dynamic programming, can be naturally seen as composition in a category of optics, and intuitively, the optimal value function is the limit of a chain of optic compositions. We illustrate this with three classic examples: the gridworld, the inverted pendulum and the savings problem. This is a first step towards a complete account of reinforcement learning in terms of parametrised optics.