# Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1909.03198 · HTML: https://mlsanity.com/paper/1909.03198/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2019.

## Built on by (most cited first, 20 of 20)

- [Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow](https://mlsanity.com/paper/2405.13629/) (2024)
- [EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control](https://mlsanity.com/paper/2511.15248/) (2025)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [Time-series Generation by Contrastive Imitation](https://mlsanity.com/paper/2311.01388/) (2023)
- [Measuring Progress in Deep Reinforcement Learning Sample Efficiency](https://mlsanity.com/paper/2102.04881/) (2021)
- [Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization](https://mlsanity.com/paper/2105.12991/) (2021)
- [Residual-MPPI: Online Policy Customization for Continuous Control](https://mlsanity.com/paper/2407.00898/) (2024)
- [Performative Policy Gradient: Optimality in Performative Reinforcement Learning](https://mlsanity.com/paper/2512.20576/) (2025)
- [OffCon3: What is state of the art anyway?](https://mlsanity.com/paper/2101.11331/) (2021)
- [Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratio](https://mlsanity.com/paper/2203.09809/) (2022)
- [Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimation](https://mlsanity.com/paper/2407.18143/) (2024)
- [A Bayesian Approach to Reinforcement Learning of Vision-Based Vehicular Control](https://mlsanity.com/paper/2104.03807/) (2021)
- [How to Choose a Reinforcement-Learning Algorithm](https://mlsanity.com/paper/2407.20917/) (2024)
- [Residual Policy Gradient: A Reward View of KL-regularized Objective](https://mlsanity.com/paper/2503.11019/) (2025)
- [Entropy-Aware Model Initialization for Effective Exploration in Deep Reinforcement Learning](https://mlsanity.com/paper/2108.10533/) (2021)
- [Attribute Controllable Beautiful Caucasian Face Generation by Aesthetics Driven Reinforcement Learning](https://mlsanity.com/paper/2208.04517/) (2022)
- [Deep Dive into Model-free Reinforcement Learning for Biological and Robotic Systems: Theory and Practice](https://mlsanity.com/paper/2405.11457/) (2024)
- [Behavior evolution-inspired approach to walking gait reinforcement training for quadruped robots](https://mlsanity.com/paper/2409.16862/) (2024)
- [DSAC-C: Constrained Maximum Entropy for Robust Discrete Soft-Actor Critic](https://mlsanity.com/paper/2310.17173/) (2023)
- [Flow Perturbation++: Multi-Step Unbiased Jacobian Estimation for High-Dimensional Boltzmann Sampling](https://mlsanity.com/paper/2601.21177/) (2026)

## Abstract (arXiv)

Maximum entropy deep reinforcement learning (RL) methods have been demonstrated on a range of challenging continuous tasks. However, existing methods either suffer from severe instability when training on large off-policy data or cannot scale to tasks with very high state and action dimensionality such as 3D humanoid locomotion. Besides, the optimality of desired Boltzmann policy set for non-optimal soft value function is not persuasive enough. In this paper, we first derive soft policy gradient based on entropy regularized expected reward objective for RL with continuous actions. Then, we present an off-policy actor-critic, model-free maximum entropy deep RL algorithm called deep soft policy gradient (DSPG) by combining soft policy gradient with soft Bellman equation. To ensure stable learning while eliminating the need of two separate critics for soft value functions, we leverage double sampling approach to making the soft Bellman equation tractable. The experimental results demonstrate that our method outperforms in performance over off-policy prior methods.