# Trust-PCL: An Off-Policy Trust Region Method for Continuous Control

arXiv: https://arxiv.org/abs/1707.01891 · HTML: https://mlsanity.com/paper/1707.01891/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2017.

## Built on by (most cited first, 30 of 30)

- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [Efficient (Soft) Q-Learning for Text Generation with Limited Good Data](https://mlsanity.com/paper/2106.07704/) (2021)
- [PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2305.15669/) (2023)
- [A Max-Min Entropy Framework for Reinforcement Learning](https://mlsanity.com/paper/2106.10517/) (2021)
- [Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learning](https://mlsanity.com/paper/2110.10719/) (2021)
- [Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization](https://mlsanity.com/paper/2006.13258/) (2020)
- [Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality](https://mlsanity.com/paper/2102.11866/) (2021)
- [Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithm](https://mlsanity.com/paper/2206.00833/) (2022)
- [Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation](https://mlsanity.com/paper/2106.04096/) (2021)
- [Divergence-Regularized Multi-Agent Actor-Critic](https://mlsanity.com/paper/2110.00304/) (2021)
- [Implicitly Regularized RL with Implicit Q-Values](https://mlsanity.com/paper/2108.07041/) (2021)
- [Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization](https://mlsanity.com/paper/2511.20718/) (2025)
- [Measuring Progress in Deep Reinforcement Learning Sample Efficiency](https://mlsanity.com/paper/2102.04881/) (2021)
- [Quasi-optimal Reinforcement Learning with Continuous Actions](https://mlsanity.com/paper/2301.08940/) (2023)
- [From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven Learning in Artificial Intelligence Tasks](https://mlsanity.com/paper/2201.08300/) (2022)
- [Dealing with Non-Stationarity in MARL via Trust-Region Decomposition](https://mlsanity.com/paper/2102.10616/) (2021)
- [Relative Trajectory Balance is equivalent to Trust-PCL](https://mlsanity.com/paper/2509.01632/) (2025)
- [Cautious Actor-Critic](https://mlsanity.com/paper/2107.05217/) (2021)
- [An Analytical Update Rule for General Policy Optimization](https://mlsanity.com/paper/2112.02045/) (2021)
- [Accelerating Primal-dual Methods for Regularized Markov Decision Processes](https://mlsanity.com/paper/2202.10506/) (2022)
- [Geometric Value Iteration: Dynamic Error-Aware KL Regularization for Reinforcement Learning](https://mlsanity.com/paper/2107.07659/) (2021)
- [Count-Based Temperature Scheduling for Maximum Entropy Reinforcement Learning](https://mlsanity.com/paper/2111.14204/) (2021)
- [Revisiting Estimation Bias in Policy Gradients for Deep Reinforcement Learning](https://mlsanity.com/paper/2301.08442/) (2023)
- [Efficient Off-Policy Learning for High-Dimensional Action Spaces](https://mlsanity.com/paper/2403.04453/) (2024)
- [Enhancing Diffusion Model Guidance through Calibration and Regularization](https://mlsanity.com/paper/2511.05844/) (2025)
- [Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning](https://mlsanity.com/paper/2107.05798/) (2021)
- [Bayesian regularization of empirical MDPs](https://mlsanity.com/paper/2208.02362/) (2022)
- [A Survey on Applications of Reinforcement Learning in Spatial Resource Allocation](https://mlsanity.com/paper/2403.03643/) (2024)
- [Trust Regions Sell, But Who's Buying? Overlap Geometry as an Alternative Trust Region for Policy Optimization](https://mlsanity.com/paper/2602.06627/) (2026)
- [RL Post-Training Diffusion Language Models Along Their Own Generation Paths](https://mlsanity.com/paper/2610.17266/) (2026)

## Abstract (arXiv)

Trust region methods, such as TRPO, are often used to stabilize policy optimization algorithms in reinforcement learning (RL). While current trust region strategies are effective for continuous control, they typically require a prohibitively large amount of on-policy interaction with the environment. To address this problem, we propose an off-policy trust region method, Trust-PCL. The algorithm is the result of observing that the optimal policy and state values of a maximum reward objective with a relative-entropy regularizer satisfy a set of multi-step pathwise consistencies along any path. Thus, Trust-PCL is able to maintain optimization stability while exploiting off-policy data to improve sample efficiency. When evaluated on a number of continuous control tasks, Trust-PCL improves the solution quality and sample efficiency of TRPO.