mlsanityData to 7 Oct 2026

Phasic Policy Gradient

arXiv 2009.04416 ↗First posted 9 Sep 2020cs.LG stat.MLcited by 121

Not on the map. Why?

Abstract (arXiv)

We introduce Phasic Policy Gradient (PPG), a reinforcement learning framework which modifies traditional on-policy actor-critic methods by separating policy and value function training into distinct phases. In prior methods, one must choose between using a shared network or separate networks to represent the policy and value function. Using separate networks avoids interference between objectives, while using a shared network allows useful features to be shared. PPG is able to achieve the best of both worlds by splitting optimization into two phases, one that advances training and one that distills features. PPG also enables the value function to be more aggressively optimized with a higher level of sample reuse. Compared to PPO, we find that PPG significantly improves sample efficiency on the challenging Procgen Benchmark.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 24; 2023: 24; 2024: 17; 2025: 21; 2026: 17.

121 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  4. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  5. 2021Learning to drive from a world on railscited by 114
  6. 2022Broken Neural Scaling Lawscited by 102
  7. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  8. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  9. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  10. 2021Scaling Scaling Laws with Board Gamescited by 62
  11. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  12. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
38 more
  1. 2025Pre-Trained Video Generative Models as World Simulatorscited by 43
  2. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  3. 2023Scaling laws for single-agent reinforcement learningcited by 39
  4. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
  5. 2021Efficient Transformers in Reinforcement Learning using Actor-Learner Distillationcited by 35
  6. 2021Batch size-invariance for policy optimizationcited by 33
  7. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  8. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
  9. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
  10. 2022Hyperbolic Deep Reinforcement Learningcited by 25
  11. 2021Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmarkcited by 23
  12. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 23
  13. 2021Cross-Trajectory Representation Learning for Zero-Shot Generalization in RLcited by 20
  14. 2023Explore to Generalize in Zero-Shot RLcited by 20
  15. 2022Improving Policy Optimization with Generalist-Specialist Learningcited by 19
  16. 2023CFlowNets: Continuous Control with Generative Flow Networkscited by 19
  17. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learningcited by 17
  18. 2022Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learningcited by 16
  19. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
  20. 2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
  21. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  22. 2025Reevaluating Policy Gradient Methods for Imperfect-Information Gamescited by 13
  23. 2024Optimal Defender Strategies for CAGE-2 using Causal Modeling and Tree Searchcited by 12
  24. 2025Dyn-O: Building Structured World Models with Object-Centric Representationscited by 12
  25. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 11
  26. 2022ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representationcited by 11
  27. 2022Reinforcement Learning Algorithms: An Overview and Classificationcited by 10
  28. 2023Guide Your Agent with Adaptive Multimodal Rewardscited by 10
  29. 2021Model-free Policy Learning with Reward Gradientscited by 9
  30. 2023Correcting discount-factor mismatch in on-policy policy gradient methodscited by 9
  31. 2021Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRLcited by 8
  32. 2022Rethinking Value Function Learning for Generalization in Reinforcement Learningcited by 8
  33. 2024A Survey Analyzing Generalization in Deep Reinforcement Learningcited by 8
  34. 2023DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewardscited by 7
  35. 2023Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Trickscited by 7
  36. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
  37. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 7
  38. 2021Learning to Design and Construct Bridge without Blueprintcited by 6

50 most cited of 121 All 121 (JSON)