mlsanityData to 7 Oct 2026

Proximal Policy Optimization Algorithms

arXiv 1707.06347 ↗First posted 20 Jul 2017cs.LGcited by 14,441

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.

Key facts. Cited by 14,441 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 0 of its 14 references that are in the map. First posted on arXiv 20 Jul 2017.

Builds on

0 of 14 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 18; 2019: 31; 2020: 50; 2021: 1016; 2022: 1132; 2023: 1633; 2024: 2475; 2025: 4109; 2026: 3977.

14,441 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
38 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  11. 2023Mastering Diverse Domains through World Modelscited by 1,193
  12. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  13. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  14. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  15. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  16. 2020Mastering Atari with Discrete World Modelscited by 990
  17. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  18. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  19. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  20. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  21. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  22. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  23. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  24. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  25. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  26. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  27. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  28. 2025Group Sequence Policy Optimizationcited by 788
  29. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  30. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  31. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  32. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  33. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  34. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  35. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  36. 2019Solving Rubik's Cube with a Robot Handcited by 699
  37. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  38. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672

50 most cited of 14,441 All 14,441 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025PPO-BR: Dual-Signal Entropy-Reward Adaptation for Trust Region Policy Optimization
  2. 2020Guided Exploration with Proximal Policy Optimization using a Single Demonstration
  3. 2025Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps
  4. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learning
  5. 2023Score Regularized Policy Optimization through Diffusion Behavior
  6. 2020POPO: Pessimistic Offline Policy Optimization
  7. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RL
  8. 2026Learning from the Near Future: Temporal Self-Distillation for RLVR