# Ensuring Monotonic Policy Improvement in Entropy-regularized Value-based Reinforcement Learning

arXiv: https://arxiv.org/abs/2008.10806 · HTML: https://mlsanity.com/paper/2008.10806/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Aug 2020.

## Built on by (most cited first, 3 of 3)

- [Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences](https://mlsanity.com/paper/2107.08285/) (2021)
- [Cautious Actor-Critic](https://mlsanity.com/paper/2107.05217/) (2021)
- [An Analytical Update Rule for General Policy Optimization](https://mlsanity.com/paper/2112.02045/) (2021)

## Abstract (arXiv)

This paper aims to establish an entropy-regularized value-based reinforcement learning method that can ensure the monotonic improvement of policies at each policy update. Unlike previously proposed lower-bounds on policy improvement in general infinite-horizon MDPs, we derive an entropy-regularization aware lower bound. Since our bound only requires the expected policy advantage function to be estimated, it is scalable to large-scale (continuous) state-space problems. We propose a novel reinforcement learning algorithm that exploits this lower-bound as a criterion for adjusting the degree of a policy update for alleviating policy oscillation. We demonstrate the effectiveness of our approach in both discrete-state maze and continuous-state inverted pendulum tasks using a linear function approximator for value estimation.