# EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization

arXiv: https://arxiv.org/abs/2110.13373 · HTML: https://mlsanity.com/paper/2110.13373/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 17 references that are in the map.
- First posted on arXiv 26 Oct 2021.

## Builds on (most cited first, 13 of 13)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPs](https://mlsanity.com/paper/1909.02769/) (2019)
- [Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1802.08365/) (2018)
- [Discount Factor as a Regularizer in Reinforcement Learning](https://mlsanity.com/paper/2007.02040/) (2020)
- [Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach](https://mlsanity.com/paper/1902.02893/) (2019)
- [Multi-Agent Trust Region Policy Optimization](https://mlsanity.com/paper/2010.07916/) (2020)
- [Robotic Table Tennis with Model-Free Reinforcement Learning](https://mlsanity.com/paper/2003.14398/) (2020)
- [Boosting Trust Region Policy Optimization by Normalizing Flows Policy](https://mlsanity.com/paper/1809.10326/) (2018)
- [Quasi-Newton Trust Region Policy Optimization](https://mlsanity.com/paper/1912.11912/) (2019)
- [Model-free Reinforcement Learning for Non-stationary Mean Field Games](https://mlsanity.com/paper/2004.02073/) (2020)
- [Model-Free Reinforcement Learning for Stochastic Games with Linear Temporal Logic Objectives](https://mlsanity.com/paper/2010.01050/) (2020)

## Built on by (most cited first, 2 of 2)

- [Hierarchical Multi-Agent DRL Based Dynamic Cluster Reconfiguration for UAV Mobility Management](https://mlsanity.com/paper/2412.16167/) (2024)
- [Quality-constrained Entropy Maximization Policy Optimization for LLM Diversity](https://mlsanity.com/paper/2602.15894/) (2026)

## Abstract (arXiv)

Trust Region Policy Optimization (TRPO) is a popular and empirically successful policy search algorithm in reinforcement learning (RL). It iteratively solved the surrogate problem which restricts consecutive policies to be close to each other. TRPO is an on-policy algorithm. On-policy methods bring many benefits, like the ability to gauge each resulting policy. However, they typically discard all the knowledge about the policies which existed before. In this work, we use a replay buffer to borrow from the off-policy learning setting to TRPO. Entropy regularization is usually used to improve policy optimization in reinforcement learning. It is thought to aid exploration and generalization by encouraging more random policy choices. We add an Entropy regularization term to advantage over π, accumulated over time steps, in TRPO. We call this update EnTRPO. Our experiments demonstrate EnTRPO achieves better performance for controlling a Cart-Pole system compared with the original TRPO