mlsanityData to 7 Oct 2026

Improving Deep Policy Gradients with Value Function Search

arXiv 2302.10145 ↗First posted 20 Feb 2023cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Deep Policy Gradient (PG) algorithms employ value networks to drive the learning of parameterized policies and reduce the variance of the gradient estimates. However, value function approximation gets stuck in local optima and struggles to fit the actual return, limiting the variance reduction efficacy and leading policies to sub-optimal performance. This paper focuses on improving value approximation and analyzing the effects on Deep PG primitives such as value prediction, variance reduction, and correlation of gradient estimates with the true gradient. To this end, we introduce a Value Function Search that employs a population of perturbed value networks to search for a better approximation. Our framework does not require additional environment interactions, gradient computations, or ensembles, providing a computationally inexpensive approach to enhance the supervised learning task on which value networks train. Crucially, we show that improving Deep PG primitives results in improved sample efficiency and policies with higher returns using common continuous control benchmark domains.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 20 Feb 2023.

Builds on

21 of 37 references.

References · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016OpenAI Gymcited by 1,684
  7. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  8. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  9. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  10. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  11. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  12. 2018CEM-RL: Combining evolutionary and gradient-based methods for policy searchcited by 53
9 more
  1. 2018GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithmscited by 40
  2. 2019A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithmscited by 26
  3. 2021Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learningcited by 26
  4. 2019Proximal Distilled Evolutionary Reinforcement Learningcited by 24
  5. 2022Combining Evolution and Deep Reinforcement Learning for Policy Search: a Surveycited by 20
  6. 2017Safe Mutations for Deep and Recurrent Neural Networks through Output Gradientscited by 14
  7. 2021Centralizing State-Values in Dueling Networks for Multi-Robot Reinforcement Learning Mapless Navigationcited by 13
  8. 2021Benchmarking Safe Deep Reinforcement Learning in Aquatic Navigationcited by 10
  9. 2021MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learningcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 2.

7 papers.

  1. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 23
  2. 2025RL2Grid: Benchmarking Reinforcement Learning in Power Grid Operationscited by 8
  3. 2024Analyzing Adversarial Inputs in Deep Reinforcement Learningcited by 7
  4. 2024Improving Policy Optimization via varepsilon-Retraincited by 5
  5. 2024On Stateful Value Factorization in Multi-Agent Reinforcement Learningcited by 5
  6. 2024Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learningcited by 1
  7. 2025Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Spacecited by 0