# SOAC: The Soft Option Actor-Critic Architecture

arXiv: https://arxiv.org/abs/2006.14363 · HTML: https://mlsanity.com/paper/2006.14363/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2020.

## Built on by (most cited first, 3 of 3)

- [Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2306.08388/) (2023)
- [Learning Skills to Navigate without a Master: A Sequential Multi-Policy Reinforcement Learning Algorithm](https://mlsanity.com/paper/2102.00168/) (2021)
- [Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPs](https://mlsanity.com/paper/2507.16473/) (2025)

## Abstract (arXiv)

The option framework has shown great promise by automatically extracting temporally-extended sub-tasks from a long-horizon task. Methods have been proposed for concurrently learning low-level intra-option policies and high-level option selection policy. However, existing methods typically suffer from two major challenges: ineffective exploration and unstable updates. In this paper, we present a novel and stable off-policy approach that builds on the maximum entropy model to address these challenges. Our approach introduces an information-theoretical intrinsic reward for encouraging the identification of diverse and effective options. Meanwhile, we utilize a probability inference model to simplify the optimization problem as fitting optimal trajectories. Experimental results demonstrate that our approach significantly outperforms prior on-policy and off-policy methods in a range of Mujoco benchmark tasks while still providing benefits for transfer learning. In these tasks, our approach learns a diverse set of options, each of whose state-action space has strong coherence.