# Wasserstein Barycenter Soft Actor-Critic

arXiv: https://arxiv.org/abs/2506.10167 · HTML: https://mlsanity.com/paper/2506.10167/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 28 of 28)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/2007.04938/) (2020)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control](https://mlsanity.com/paper/2405.16158/) (2024)
- [Better Exploration with Optimistic Actor-Critic](https://mlsanity.com/paper/1910.12807/) (2019)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Tactical Optimism and Pessimism for Deep Reinforcement Learning](https://mlsanity.com/paper/2102.03765/) (2021)
- [Softmax Deep Double Deterministic Policy Gradients](https://mlsanity.com/paper/2010.09177/) (2020)
- [Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2104.09122/) (2021)
- [Efficient Continuous Control with Double Actors and Regularized Critics](https://mlsanity.com/paper/2106.03050/) (2021)
- [Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2101.09458/) (2021)
- [ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search](https://mlsanity.com/paper/1811.02696/) (2018)
- [Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guarantees](https://mlsanity.com/paper/2301.13375/) (2023)
- [Continuous Control With Ensemble Deep Deterministic Policy Gradients](https://mlsanity.com/paper/2111.15382/) (2021)
- [Risk-Aware Reinforcement Learning through Optimal Transport Theory](https://mlsanity.com/paper/2309.06239/) (2023)
- [Understanding Reward Ambiguity Through Optimal Transport Theory in Inverse Reinforcement Learning](https://mlsanity.com/paper/2310.12055/) (2023)
- [Value Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradients](https://mlsanity.com/paper/2112.11216/) (2021)
- [The Synergy Between Optimal Transport Theory and Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2401.10949/) (2024)
- [Wasserstein Actor-Critic: Directed Exploration via Optimism for Continuous-Actions Control](https://mlsanity.com/paper/2303.02378/) (2023)
- [Optimal Transport-Assisted Risk-Sensitive Q-Learning](https://mlsanity.com/paper/2406.11774/) (2024)
- [Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learning](https://mlsanity.com/paper/2501.10605/) (2025)
- [Efficient Reinforcement Learning via Decoupling Exploration and Utilization](https://mlsanity.com/paper/2312.15965/) (2023)
- [Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learning](https://mlsanity.com/paper/2409.19231/) (2024)

## Built on by (most cited first, 1 of 1)

- [Can Optimal Transport Improve Federated Inverse Reinforcement Learning?](https://mlsanity.com/paper/2601.00309/) (2026)

## Abstract (arXiv)

Deep off-policy actor-critic algorithms have emerged as the leading framework for reinforcement learning in continuous control domains. However, most of these algorithms suffer from poor sample efficiency, especially in environments with sparse rewards. In this paper, we take a step towards addressing this issue by providing a principled directed exploration strategy. We propose Wasserstein Barycenter Soft Actor-Critic (WBSAC) algorithm, which benefits from a pessimistic actor for temporal difference learning and an optimistic actor to promote exploration. This is achieved by using the Wasserstein barycenter of the pessimistic and optimistic policies as the exploration policy and adjusting the degree of exploration throughout the learning process. We compare WBSAC with state-of-the-art off-policy actor-critic algorithms and show that WBSAC is more sample-efficient on MuJoCo continuous control tasks.