mlsanityData to 14 Oct 2026

Transductive Off-policy Proximal Policy Optimization

arXiv 2406.03894 ↗First posted 6 Jun 2024cs.LGcited by 5

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Proximal Policy Optimization (PPO) is a popular model-free reinforcement learning algorithm, esteemed for its simplicity and efficacy. However, due to its inherent on-policy nature, its proficiency in harnessing data from disparate policies is constrained. This paper introduces a novel off-policy extension to the original PPO method, christened Transductive Off-policy PPO (ToPPO). Herein, we provide theoretical justification for incorporating off-policy data in PPO training and prudent guidelines for its safe application. Our contribution includes a novel formulation of the policy improvement lower bound for prospective policies derived from off-policy data, accompanied by a computationally efficient mechanism to optimize this bound, underpinned by assurances of monotonic improvement. Comprehensive experimental results across six representative tasks underscore ToPPO's promising performance.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 16 of its 30 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

16 of 30 references.

References · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2017Constrained Policy Optimizationcited by 983
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  8. 2017Deep Reinforcement Learning that Matterscited by 754
  9. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  10. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  11. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  12. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 40
4 more
  1. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
  2. 2019Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Targetcited by 31
  3. 2019Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learningcited by 12
  4. 2016Convergence analysis of the direct extension of ADMM for multiple-block separable convex minimizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Trainingcited by 34
  2. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  3. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimizationcited by 1
  4. 2025Representation Convergence: Mutual Distillation is Secretly a Form of Regularizationcited by 1
  5. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithm
  2. 2024Beyond the Boundaries of Proximal Policy Optimization
  3. 2026Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?
  4. 2024Simple Policy Optimization
  5. 2023Behavior Proximal Policy Optimization
  6. 2025Truncated Proximal Policy Optimization
  7. 2026KLip-PPO: A per-sample KL perspective on PPO-Clip
  8. 2020POPO: Pessimistic Offline Policy Optimization