mlsanityData to 14 Oct 2026

Generalized Proximal Policy Optimization with Sample Reuse

arXiv 2111.00072 ↗First posted 29 Oct 2021cs.LG cs.AI stat.MLcited by 40

Not on the map. Why?

Abstract (arXiv)

In real-world decision making tasks, it is critical for data-driven reinforcement learning methods to be both stable and sample efficient. On-policy methods typically generate reliable policy improvement throughout training, while off-policy methods make more efficient use of data through sample reuse. In this work, we combine the theoretically supported stability benefits of on-policy algorithms with the sample efficiency of off-policy algorithms. We develop policy improvement guarantees that are suitable for the off-policy setting, and connect these bounds to the clipping mechanism used in Proximal Policy Optimization. This motivates an off-policy version of the popular algorithm that we call Generalized Proximal Policy Optimization with Sample Reuse. We demonstrate both theoretically and empirically that our algorithm delivers improved performance by effectively balancing the competing goals of stability and sample efficiency.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 20 of its 25 references that are in the map. First posted on arXiv 29 Oct 2021.

Builds on

20 of 25 references.

References · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2016OpenAI Gymcited by 1,736
  8. 2017Constrained Policy Optimizationcited by 983
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  10. 2017Deep Reinforcement Learning that Matterscited by 754
  11. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  12. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
8 more
  1. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 81
  2. 2019Truly Proximal Policy Optimizationcited by 69
  3. 2017Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learningcited by 40
  4. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  5. 2019Trust Region-Guided Proximal Policy Optimizationcited by 36
  6. 2016Combining policy gradient and Q-learningcited by 24
  7. 2019Combining Experience Replay with Exploration by Random Network Distillationcited by 5
  8. 2020Continuous-action Reinforcement Learning for Playing Racing Games: Comparing SPG to PPOcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 8; 2024: 7; 2025: 8; 2026: 10.

40 papers.

Sort:

By area:

  1. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 56
  2. 2023Behavior Proximal Policy Optimizationcited by 43
  3. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Trainingcited by 34
  4. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  5. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuningcited by 30
  6. 2023Order Matters: Agent-by-agent Policy Optimizationcited by 24
  7. 2022You May Not Need Ratio Clipping in PPOcited by 19
  8. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Modelcited by 14
  9. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
  10. 2022Self-Adaptive Driving in Nonstationary Environments through Conjectural Online Lookahead Adaptationcited by 12
  11. 2023Adversarial Imitation Learning from Visual Observations using Latent Informationcited by 11
  12. 2025Truncated Proximal Policy Optimizationcited by 8
28 more
  1. 2024Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rankcited by 7
  2. 2023On-Policy Policy Gradient Reinforcement Learning Without On-Policy Samplingcited by 6
  3. 2025Disentangling Length Bias In Preference Learning Via Response-Conditioned Modelingcited by 6
  4. 2024Transductive Off-policy Proximal Policy Optimizationcited by 5
  5. 2022Combining imitation and deep reinforcement learning to accomplish human-level performance on a virtual foraging taskcited by 4
  6. 2022Opportunities and Challenges from Using Animal Videos in Reinforcement Learning for Navigationcited by 4
  7. 2025Enhancing PPO with Trajectory-Aware Hybrid Policiescited by 4
  8. 2025PPO in the Fisher-Rao geometrycited by 4
  9. 2023Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimizationcited by 3
  10. 2024Reflective Policy Optimizationcited by 3
  11. 2026Clipping-Free Policy Optimization for Large Language Modelscited by 3
  12. 2023Absolute Policy Optimizationcited by 2
  13. 2026SymbXRL: Symbolic Explainable Deep Reinforcement Learning for Mobile Networkscited by 2
  14. 2026Staleness-Learning Rate Scaling Laws for Asynchronous RLHFcited by 2
  15. 2022Safe adaptation in multiagent competitioncited by 1
  16. 2022Generalized Policy Improvement Algorithms with Theoretically Supported Sample Reusecited by 1
  17. 2023Dropout Strategy in Reinforcement Learning: Limiting the Surrogate Objective Variance in Policy Optimization Methodscited by 1
  18. 2024A Hierarchical DRL Approach for Resource Optimization in Multi-RIS Multi-Operator Networkscited by 1
  19. 2026Align and Filter: Improving Performance in Asynchronous On-Policy RLcited by 1
  20. 2023Deep Metric Tensor Regularized Policy Gradientcited by 0
  21. 2024A Model-Based Approach for Improving Reinforcement Learning Efficiency Leveraging Expert Observationscited by 0
  22. 2024Do Agents Dream of Electric Sheep?: Improving Generalization in Reinforcement Learning through Generative Learningcited by 0
  23. 2024CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Communitycited by 0
  24. 2026ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithmcited by 0
  25. 2026Match or Replay: Self Imitating Proximal Policy Optimizationcited by 0
  26. 2026Not All Transitions Matter: Evidence from PPOcited by 0
  27. 2026RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agentscited by 0
  28. 2026Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?cited by 0