mlsanityData to 7 Oct 2026

Trust-PCL: An Off-Policy Trust Region Method for Continuous Control

arXiv 1707.01891 ↗First posted 6 Jul 2017cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

Trust region methods, such as TRPO, are often used to stabilize policy optimization algorithms in reinforcement learning (RL). While current trust region strategies are effective for continuous control, they typically require a prohibitively large amount of on-policy interaction with the environment. To address this problem, we propose an off-policy trust region method, Trust-PCL. The algorithm is the result of observing that the optimal policy and state values of a maximum reward objective with a relative-entropy regularizer satisfy a set of multi-step pathwise consistencies along any path. Thus, Trust-PCL is able to maintain optimization stability while exploiting off-policy data to improve sample efficiency. When evaluated on a number of continuous control tasks, Trust-PCL improves the solution quality and sample efficiency of TRPO.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 14; 2022: 4; 2023: 3; 2024: 2; 2025: 3; 2026: 1.

29 papers.

Sort:

By area:

  1. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  2. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
  3. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
  4. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  5. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  6. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimizationcited by 22
  7. 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 20
  8. 2022Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithmcited by 17
  9. 2021Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximationcited by 15
  10. 2021Divergence-Regularized Multi-Agent Actor-Criticcited by 15
  11. 2021Implicitly Regularized RL with Implicit Q-Valuescited by 9
  12. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
17 more
  1. 2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalizationcited by 8
  2. 2023Quasi-optimal Reinforcement Learning with Continuous Actionscited by 7
  3. 2022From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven Learning in Artificial Intelligence Taskscited by 6
  4. 2021Dealing with Non-Stationarity in MARL via Trust-Region Decompositioncited by 5
  5. 2021An Analytical Update Rule for General Policy Optimizationcited by 3
  6. 2025Relative Trajectory Balance is equivalent to Trust-PCLcited by 3
  7. 2021Cautious Actor-Criticcited by 2
  8. 2022Accelerating Primal-dual Methods for Regularized Markov Decision Processescited by 2
  9. 2021Geometric Value Iteration: Dynamic Error-Aware KL Regularization for Reinforcement Learningcited by 1
  10. 2021Count-Based Temperature Scheduling for Maximum Entropy Reinforcement Learningcited by 1
  11. 2023Revisiting Estimation Bias in Policy Gradients for Deep Reinforcement Learningcited by 1
  12. 2024Efficient Off-Policy Learning for High-Dimensional Action Spacescited by 1
  13. 2025Enhancing Diffusion Model Guidance through Calibration and Regularizationcited by 1
  14. 2021Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learningcited by 0
  15. 2022Bayesian regularization of empirical MDPscited by 0
  16. 2024A Survey on Applications of Reinforcement Learning in Spatial Resource Allocationcited by 0
  17. 2026Trust Regions Sell, But Who's Buying? Overlap Geometry as an Alternative Trust Region for Policy Optimizationcited by 0