# Reduce, Reuse, Recycle: Categories for Compositional Reinforcement Learning

arXiv: https://arxiv.org/abs/2408.13376 · HTML: https://mlsanity.com/paper/2408.13376/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 23 Aug 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 25 of 25)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer](https://mlsanity.com/paper/1609.07088/) (2016)
- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [A Boolean Task Algebra for Reinforcement Learning](https://mlsanity.com/paper/2001.01394/) (2020)
- [Categorical Foundations of Gradient-Based Learning](https://mlsanity.com/paper/2103.01931/) (2021)
- [Modular Lifelong Reinforcement Learning via Neural Composition](https://mlsanity.com/paper/2207.00429/) (2022)
- [Exploration in Structured Reinforcement Learning](https://mlsanity.com/paper/1806.00775/) (2018)
- [Environment Generation for Zero-Shot Compositional Reinforcement Learning](https://mlsanity.com/paper/2201.08896/) (2022)
- [Misspecification in Inverse Reinforcement Learning](https://mlsanity.com/paper/2212.03201/) (2022)
- [CompoSuite: A Compositional Reinforcement Learning Benchmark](https://mlsanity.com/paper/2207.04136/) (2022)
- [Solving Compositional Reinforcement Learning Problems via Task Reduction](https://mlsanity.com/paper/2103.07607/) (2021)
- [Representable Markov Categories and Comparison of Statistical Experiments in Categorical Probability](https://mlsanity.com/paper/2010.07416/) (2020)
- [Co-Design of Embodied Intelligence: A Structured Approach](https://mlsanity.com/paper/2011.10756/) (2020)
- [Co-Design of Autonomous Systems: From Hardware Selection to Control Synthesis](https://mlsanity.com/paper/2011.10758/) (2020)
- [Constructing MDP Abstractions Using Data with Formal Guarantees](https://mlsanity.com/paper/2206.14402/) (2022)
- [Categorical semantics of compositional reinforcement learning](https://mlsanity.com/paper/2208.13687/) (2022)
- [Categorical Semantics of Cyber-Physical Systems Theory](https://mlsanity.com/paper/2010.08003/) (2020)
- [Compositional Probabilistic Model Checking with String Diagrams of MDPs](https://mlsanity.com/paper/2307.08765/) (2023)
- [Reinforcement Learning in Categorical Cybernetics](https://mlsanity.com/paper/2404.02688/) (2024)
- [Co-Design to Enable User-Friendly Tools to Assess the Impact of Future Mobility Solutions](https://mlsanity.com/paper/2008.08975/) (2020)
- [Compositional Thinking in Cyberphysical Systems Theory](https://mlsanity.com/paper/2105.12911/) (2021)
- [A Compositional Framework for First-Order Optimization](https://mlsanity.com/paper/2403.05711/) (2024)
- [Compositional Cyber-Physical Systems Modeling](https://mlsanity.com/paper/2101.10484/) (2021)

## Built on by (most cited first, 2 of 2)

- [State Combinatorial Generalization In Decision Making With Conditional Diffusion Models](https://mlsanity.com/paper/2501.13241/) (2025)
- [Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2606.16515/) (2026)

## Abstract (arXiv)

In reinforcement learning, conducting task composition by forming cohesive, executable sequences from multiple tasks remains challenging. However, the ability to (de)compose tasks is a linchpin in developing robotic systems capable of learning complex behaviors. Yet, compositional reinforcement learning is beset with difficulties, including the high dimensionality of the problem space, scarcity of rewards, and absence of system robustness after task composition. To surmount these challenges, we view task composition through the prism of category theory -- a mathematical discipline exploring structures and their compositional relationships. The categorical properties of Markov decision processes untangle complex tasks into manageable sub-tasks, allowing for strategical reduction of dimensionality, facilitating more tractable reward structures, and bolstering system robustness. Experimental results support the categorical theory of reinforcement learning by enabling skill reduction, reuse, and recycling when learning complex robotic arm tasks.