Data to 7 Oct 2026

Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO

arXiv 2005.12729 ↗First posted 25 May 2020cs.LG cs.RO stat.MLcited by 184

Not on the map. Why?

Cited by 54 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, we investigate the consequences of "code-level optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations turn out to have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function. These insights show the difficulty and importance of attributing performance gains in deep reinforcement learning. Code for reproducing our results is available at https://github.com/MadryLab/implementation-matters .

Key facts. Cited by 184 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 17; 2022: 16; 2023: 30; 2024: 42; 2025: 37; 2026: 41.

184 papers.

Sort:

By area:

2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
2023Mitigating the Alignment Tax of RLHFcited by 181
2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133

38 more

2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
2021Learning to Iteratively Solve Routing Problems with Dual-Aspect Collaborative Transformercited by 108
2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 82
2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
2024Simplifying Deep Temporal Difference Learningcited by 71
2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspectivecited by 69
2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 58
2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoningcited by 53
2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
2021Differentiable Trust Region Layers for Deep Reinforcement Learningcited by 24
2024Streaming Deep Reinforcement Learning Finally Workscited by 24
2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
2022You May Not Need Ratio Clipping in PPOcited by 19
2022Deep Black-Box Reinforcement Learning with Movement Primitivescited by 18
2021Hyperparameter Selection for Imitation Learningcited by 17
2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
2022Policy-Based Bayesian Experimental Design for Non-Differentiable Implicit Modelscited by 15
2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
2024Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Expertscited by 14
2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14

50 most cited of 184 All 184 (JSON)