# Asynchronous Coagent Networks

arXiv: https://arxiv.org/abs/1902.05650 · HTML: https://mlsanity.com/paper/1902.05650/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2019.

## Built on by (most cited first, 4 of 4)

- [Discovery of Options via Meta-Learned Subgoals](https://mlsanity.com/paper/2102.06741/) (2021)
- [When Can Model-Free Reinforcement Learning be Enough for Thinking?](https://mlsanity.com/paper/2506.17124/) (2025)
- [Edge-Compatible Reinforcement Learning for Recommendations](https://mlsanity.com/paper/2112.05812/) (2021)
- [Coagent Networks: Generalized and Scaled](https://mlsanity.com/paper/2305.09838/) (2023)

## Abstract (arXiv)

Coagent policy gradient algorithms (CPGAs) are reinforcement learning algorithms for training a class of stochastic neural networks called coagent networks. In this work, we prove that CPGAs converge to locally optimal policies. Additionally, we extend prior theory to encompass asynchronous and recurrent coagent networks. These extensions facilitate the straightforward design and analysis of hierarchical reinforcement learning algorithms like the option-critic, and eliminate the need for complex derivations of customized learning rules for these algorithms.