mlsanityData to 7 Oct 2026

Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying

arXiv 2606.00151 ↗First posted 29 May 2026cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning (RL), agents benefit from exploration only because they repeatedly encounter similar states: trying different actions can improve performance or reduce uncertainty; without such retries, a greedy policy is optimal. We formalize this intuition with ReMax, an objective that evaluates a policy by the expected maximum return over M samples, where M is a positive integer, while accounting for return uncertainty. Optimizing this objective induces stochastic exploration as an emergent property, without explicit bonus terms. For efficient policy optimization, we derive a new policy-gradient formulation for ReMax and introduce ReMax PPO (RePPO), a PPO variant that optimizes ReMax while generalizing the discrete retry count M to a continuous parameter m > 0, enabling fine-grained control of exploration. Empirically, RePPO promotes exploration, without any explicit exploration bonuses, on the MinAtar and Craftax benchmarks. The code is available at https://github.com/nissymori/remax-rl .

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 66 of its 99 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

66 of 99 references.

References · 66

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2018Exploration by Random Network Distillationcited by 966
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  12. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
38 more
  1. 2016Deep Exploration via Bootstrapped DQNcited by 518
  2. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  3. 2018Is Q-learning Provably Efficient?cited by 485
  4. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  6. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  8. 2020First return, then explorecited by 256
  9. 2017Count-Based Exploration with Neural Density Modelscited by 255
  10. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  11. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  12. 2020The NetHack Learning Environmentcited by 196
  13. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  14. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  15. 2017Deep Exploration via Randomized Value Functionscited by 155
  16. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  17. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  18. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
  19. 2018Episodic Curiosity through Reachabilitycited by 109
  20. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
  21. 2022Discovered Policy Optimisationcited by 103
  22. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  23. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
  24. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  25. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
  26. 2024Simplifying Deep Temporal Difference Learningcited by 71
  27. 2020Munchausen Reinforcement Learningcited by 68
  28. 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
  29. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 58
  30. 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
  31. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 49
  32. 2023Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learningcited by 45
  33. 2017EX2: Exploration with Exemplar Models for Deep Reinforcement Learningcited by 42
  34. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
  35. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 40
  36. 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
  37. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  38. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learningcited by 2
  2. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2
  3. 2026Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generationcited by 2
  4. 2026Retry Policy Gradients in Continuous Action Spacescited by 1
  5. 2026OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimationcited by 1