mlsanityData to 7 Oct 2026

Behind the Myth of Exploration in Policy Gradients

arXiv 2402.00162 ↗First posted 31 Jan 2024cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

In order to compute near-optimal policies with policy-gradient algorithms, it is common in practice to include intrinsic exploration terms in the learning objective. Although the effectiveness of these terms is usually justified by an intrinsic need to explore environments, we propose a novel analysis with the lens of numerical optimization. Two criteria are introduced on the learning objective and two others on its stochastic gradient estimates, and are afterwards used to discuss the quality of the policy after optimization. The analysis sheds light on two separate effects of exploration techniques. First, they make it possible to smooth the learning objective and to eliminate local optima while preserving the global maximum. Second, they modify the gradient estimates, increasing the probability that the stochastic parameter updates eventually provide an optimal policy. We empirically illustrate these effects with exploration strategies based on entropy bonuses, identifying limitations and suggesting directions for future work.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

36 of 56 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  4. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  5. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  6. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  7. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  8. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  9. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  10. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  11. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  12. 2019Efficient Exploration via State Marginal Matchingcited by 186
24 more
  1. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
  2. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  3. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  4. 2018Understanding the impact of entropy on policy optimizationcited by 171
  5. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  6. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  7. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
  8. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 85
  9. 2020On the Convergence of SGD with Biased Gradientscited by 79
  10. 2020A Unifying View of Optimism in Episodic Reinforcement Learningcited by 50
  11. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  12. 2020Beyond variance reduction: Understanding the true impact of baselines on policy optimizationcited by 32
  13. 2021Geometric Entropic Explorationcited by 32
  14. 2021Regularized Policies are Reward Robustcited by 32
  15. 2020Novelty Search in Representational Space for Sample Efficient Explorationcited by 26
  16. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  17. 2018Stochastic Gradient Descent with Biased but Consistent Gradient Estimatorscited by 18
  18. 2021Understanding the Effect of Stochasticity in Policy Optimizationcited by 18
  19. 2019Marginalized State Distribution Entropy Regularization in Policy Optimizationcited by 16
  20. 2020A Deep Reinforcement Learning Framework for Continuous Intraday Market Biddingcited by 15
  21. 2022On the Hidden Biases of Policy Mirror Ascent in Continuous Action Spacescited by 11
  22. 2020Policy Gradient using Weak Derivatives for Reinforcement Learningcited by 7
  23. 2023Policy Gradient Algorithms Implicitly Optimize by Continuationcited by 4
  24. 2024Optimal Control of Renewable Energy Communities subject to Network Peak Fees with Model Predictive Control and Reinforcement Learning Algorithmscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 2.

3 papers.

  1. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measurescited by 3
  2. 2026How Log-Barrier Helps Exploration in Policy Optimizationcited by 0
  3. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measurescited by 0