mlsanityData to 14 Oct 2026

Improving Value Estimation Critically Enhances Vanilla Policy Gradient

arXiv 2505.19247 ↗First posted 25 May 2025cs.LG cs.AI cs.ROcited by 5

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Modern policy gradient algorithms, such as TRPO and PPO, outperform vanilla policy gradient in many RL tasks. Questioning the common belief that enforcing approximate trust regions leads to steady policy improvement in practice, we show that the more critical factor is the enhanced value estimation accuracy from more value update steps in each iteration. To demonstrate, we show that by simply increasing the number of value update steps per iteration, vanilla policy gradient itself can achieve performance comparable to or better than PPO in all the standard continuous control benchmark environments. Importantly, this simple change to vanilla policy gradient is significantly more robust to hyperparameter choices, opening up the possibility that RL algorithms may still become more effective and easier to use.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 40 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

23 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Mollification Effects of Policy Gradient Methods28 May

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  9. 2016OpenAI Gymcited by 1,736
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  11. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  12. 2017Deep Reinforcement Learning that Matterscited by 754
10 more
  1. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  2. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  3. 2020Phasic Policy Gradientcited by 123
  4. 2023Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learningcited by 111
  5. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  6. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 48
  7. 2019Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policycited by 43
  8. 2022You May Not Need Ratio Clipping in PPOcited by 19
  9. 2024A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learningcited by 15
  10. 2023Fractal Landscapes in Policy Optimizationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Criticcited by 30
  2. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 10
  3. 2026RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learningcited by 1
  4. 2026Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Errorcited by 1
  5. 2026Refined Analysis of Entropy-Regularized Actor-Criticcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Approximate Next Policy Sampling: Replacing Conservative Target Policy Updates in Deep RL
  2. 2025Stabilizing Policy Gradient Methods via Reward Profiling
  3. 2026Beyond Importance Sampling: Rejection-Gated Policy Optimization
  4. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations
  5. 2025Algorithm-Relative Trajectory Valuation in Policy Gradient Control
  6. 2026Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization
  7. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization
  8. 2026Delightful Policy Gradient