mlsanityData to 7 Oct 2026

What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study

arXiv 2006.05990 ↗First posted 10 Jun 2020cs.LG stat.MLcited by 157

Not on the map. Why?

Abstract (arXiv)

In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress [Engstrom'20]. As a step towards filling that gap, we implement >50 such ``choices'' in a unified on-policy RL framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on-policy training of RL agents.

Key facts. Cited by 157 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 24; 2023: 27; 2024: 34; 2025: 26; 2026: 25.

157 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  3. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  4. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  5. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  6. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  7. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  8. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 74
  9. 2021What Matters for Adversarial Imitation Learning?cited by 70
  10. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  11. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  12. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
38 more
  1. 2025Magistralcited by 57
  2. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  3. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoningcited by 53
  4. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  5. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  6. 2021Training Larger Networks for Deep Reinforcement Learningcited by 37
  7. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  8. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
  9. 2024EgoGen: An Egocentric Synthetic Data Generatorcited by 27
  10. 2025Single-stream Policy Optimizationcited by 27
  11. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learningcited by 26
  12. 2021Differentiable Trust Region Layers for Deep Reinforcement Learningcited by 24
  13. 2021Learning Representations for Pixel-based Control: What Matters and Why?cited by 24
  14. 2023Order Matters: Agent-by-agent Policy Optimizationcited by 24
  15. 2023RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learningcited by 24
  16. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  17. 2022Learning Torque Control for Quadrupedal Locomotioncited by 22
  18. 2023For Pre-Trained Vision Models in Motor Control, Not All Policy Learning Methods are Created Equalcited by 22
  19. 2023LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarioscited by 20
  20. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
  21. 2022You May Not Need Ratio Clipping in PPOcited by 19
  22. 2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 18
  23. 2021Hyperparameter Selection for Imitation Learningcited by 17
  24. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learningcited by 17
  25. 2024Natural Language Reinforcement Learningcited by 17
  26. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  27. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
  28. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 16
  29. 2025Learning coordinated badminton skills for legged manipulatorscited by 16
  30. 2024A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learningcited by 15
  31. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14
  32. 2025Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Schemecited by 14
  33. 2024FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learningcited by 13
  34. 2021Correcting Momentum in Temporal Difference Learningcited by 12
  35. 2022A Survey on Reinforcement Learning Methods in Character Animationcited by 12
  36. 2022Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Searchcited by 12
  37. 2024Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Plannercited by 12
  38. 2024Multi-Task Reinforcement Learning for Quadrotorscited by 12

50 most cited of 157 All 157 (JSON)