# Categorical semantics of compositional reinforcement learning

arXiv: https://arxiv.org/abs/2208.13687 · HTML: https://mlsanity.com/paper/2208.13687/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 88 references that are in the map.
- First posted on arXiv 29 Aug 2022.

## Builds on (most cited first, 28 of 28)

- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Bilinear Classes: A Structural Framework for Provable Generalization in RL](https://mlsanity.com/paper/2103.10897/) (2021)
- [MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning](https://mlsanity.com/paper/2006.16908/) (2020)
- [Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer](https://mlsanity.com/paper/1609.07088/) (2016)
- [AI Safety Gridworlds](https://mlsanity.com/paper/1711.09883/) (2017)
- [Compositional generalization through meta sequence-to-sequence learning](https://mlsanity.com/paper/1906.05381/) (2019)
- [LTL2Action: Generalizing LTL Instructions for Multi-Task RL](https://mlsanity.com/paper/2102.06858/) (2021)
- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [A Boolean Task Algebra for Reinforcement Learning](https://mlsanity.com/paper/2001.01394/) (2020)
- [Reward Machines for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2007.01962/) (2020)
- [Exploration in Structured Reinforcement Learning](https://mlsanity.com/paper/1806.00775/) (2018)
- [Environment Generation for Zero-Shot Compositional Reinforcement Learning](https://mlsanity.com/paper/2201.08896/) (2022)
- [The Logical Options Framework](https://mlsanity.com/paper/2102.12571/) (2021)
- [CompoSuite: A Compositional Reinforcement Learning Benchmark](https://mlsanity.com/paper/2207.04136/) (2022)
- [Solving Compositional Reinforcement Learning Problems via Task Reduction](https://mlsanity.com/paper/2103.07607/) (2021)
- [In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications](https://mlsanity.com/paper/2110.09461/) (2021)
- [Verifiable and Compositional Reinforcement Learning Systems](https://mlsanity.com/paper/2106.05864/) (2021)
- [Representable Markov Categories and Comparison of Statistical Experiments in Categorical Probability](https://mlsanity.com/paper/2010.07416/) (2020)
- [Compositionality as we see it, everywhere around us](https://mlsanity.com/paper/2110.05327/) (2021)
- [Co-Design of Embodied Intelligence: A Structured Approach](https://mlsanity.com/paper/2011.10756/) (2020)
- [Categorical Semantics of Cyber-Physical Systems Theory](https://mlsanity.com/paper/2010.08003/) (2020)
- [Algebraic Databases](https://mlsanity.com/paper/1602.03501/) (2016)
- [The algebra and machine representation of statistical models](https://mlsanity.com/paper/2006.08945/) (2020)
- [Networks of open systems](https://mlsanity.com/paper/1705.04814/) (2017)
- [Networks of hybrid open systems](https://mlsanity.com/paper/1908.10447/) (2019)
- [Formal composition of hybrid systems](https://mlsanity.com/paper/1911.01267/) (2019)
- [Multiple Model Synchronization with Multiary Delta Lenses with Amendment and K-Putput](https://mlsanity.com/paper/1911.11302/) (2019)

## Built on by (most cited first, 5 of 5)

- [From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks](https://mlsanity.com/paper/2405.15164/) (2024)
- [Reinforcement Learning in Categorical Cybernetics](https://mlsanity.com/paper/2404.02688/) (2024)
- [Reduce, Reuse, Recycle: Categories for Compositional Reinforcement Learning](https://mlsanity.com/paper/2408.13376/) (2024)
- [A Dynamical Framework for Cognitive Processes Based on Transformations and Semantic Equivalence](https://mlsanity.com/paper/2605.23942/) (2026)
- [Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning](https://mlsanity.com/paper/2606.25357/) (2026)

## Abstract (arXiv)

Compositional knowledge representations in reinforcement learning (RL) facilitate modular, interpretable, and safe task specifications. However, generating compositional models requires the characterization of minimal assumptions for the robustness of the compositionality feature, especially in the case of functional decompositions. Using a categorical point of view, we develop a knowledge representation framework for a compositional theory of RL. Our approach relies on the theoretical study of the category MDP, whose objects are Markov decision processes (MDPs) acting as models of tasks. The categorical semantics models the compositionality of tasks through the application of pushout operations akin to combining puzzle pieces. As a practical application of these pushout operations, we introduce zig-zag diagrams that rely on the compositional guarantees engendered by the category MDP. We further prove that properties of the category MDP unify concepts, such as enforcing safety requirements and exploiting symmetries, generalizing previous abstraction theories for RL.