mlsanityData to 14 Oct 2026

Retry Policy Gradients in Continuous Action Spaces

arXiv 2606.05888 ↗First posted 4 Jun 2026cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Retry-based objectives such as pass@K and max@K optimize the best return obtained from multiple sampled trajectories, and recent work has shown that they can promote exploration without explicit exploration bonuses. In discrete action spaces, ReMax was shown to do so by adapting to return uncertainty. In this work, we introduce pathwise derivative estimators for retry objectives and use them to extend ReMax to continuous action spaces. We study the resulting learning dynamics and show that, even with deterministic rewards, ReMax can encourage stochastic exploration by reshaping the policy-gradient landscape. In particular, it alters gradients both in direction, biasing updates toward higher policy entropy, and in magnitude, damping gradients and slowing convergence. We further show that Adam's adaptive normalization can mitigate this damping, depending on its numerical stabilization parameter. Empirically, we instantiate this objective as ReMax Actor-Critic (ReMAC), an off-policy actor--critic algorithm that optimizes the ReMax objective using a pathwise derivative estimator. Our experiments show that ReMAC can promote higher policy entropy without entropy regularization and achieves performance comparable to SAC.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 36 of its 46 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

36 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems21 May
  2. 2024More Efficient Randomized Exploration for Reinforcement Learning via Approximate Sampling18 Jun

Other references · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2018Exploration by Random Network Distillationcited by 993
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  8. 2014Convex Optimization: Algorithms and Complexitycited by 575
  9. 2016Deep Exploration via Bootstrapped DQNcited by 527
  10. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  11. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  12. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
22 more
  1. 2017Noisy Networks for Explorationcited by 266
  2. 2017Count-Based Exploration with Neural Density Modelscited by 257
  3. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  4. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  5. 2017Parameter Space Noise for Explorationcited by 161
  6. 2017Deep Exploration via Randomized Value Functionscited by 156
  7. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  8. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  9. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  10. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
  11. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  12. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
  13. 2018Pathwise Derivatives Beyond the Reparameterization Trickcited by 36
  14. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  15. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  16. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  17. 2018ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Searchcited by 12
  18. 2021A unified view of likelihood ratio and reparameterization gradientscited by 12
  19. 2019Total stochastic gradient algorithms and applications in reinforcement learningcited by 7
  20. 2025Polychromic Objectives for Reinforcement Learningcited by 7
  21. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  22. 2026Regret Analysis of Retry-Based Banditscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Soft Actor-Critic with Beta Policy via Implicit Reparameterization Gradients
  2. 2022Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithms
  3. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimization
  4. 2025Relative Entropy Pathwise Policy Optimization
  5. 2026Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic
  6. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization
  7. 2026Reparameterization Flow Policy Optimization
  8. 2025Sequential Policy Gradient for Adaptive Hyperparameter Optimization