mlsanityData to 14 Oct 2026

Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functions

arXiv 2410.11833 ↗First posted 15 Oct 2024cs.LG cs.AI cs.ROcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In reinforcement learning, off-policy actor-critic methods like DDPG and TD3 use deterministic policy gradients: the Q-function is learned from environment data, while the actor maximizes it via gradient ascent. We observe that in complex tasks such as dexterous manipulation and restricted locomotion with mobility constraints, the Q-function exhibits many local optima, making gradient ascent prone to getting stuck. To address this, we introduce SAVO, an actor architecture that (i) generates multiple action proposals and selects the one with the highest Q-value, and (ii) approximates the Q-function repeatedly by truncating poor local optima to guide gradient ascent more effectively. We evaluate tasks such as restricted locomotion, dexterous manipulation, and large discrete-action space recommender systems and show that our actor finds optimal actions more frequently and outperforms alternate actor architectures.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 43 of its 78 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

43 of 78 references.

References · 43

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  8. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  9. 2017Deep Setscited by 1,421
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
31 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  2. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  3. 2017Exploring Generalization in Deep Learningcited by 600
  4. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 546
  5. 2016Deep Exploration via Bootstrapped DQNcited by 527
  6. 2021Implicit Behavioral Cloningcited by 452
  7. 2016Input Convex Neural Networkscited by 414
  8. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  9. 2014The Loss Surfaces of Multilayer Networkscited by 280
  10. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  11. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  12. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 161
  13. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
  14. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 100
  15. 2019RecSim: A Configurable Simulation Platform for Recommender Systemscited by 94
  16. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  17. 2018Generative Adversarial User Model for Reinforcement Learning Based Recommendation Systemcited by 64
  18. 2018CEM-RL: Combining evolutionary and gradient-based methods for policy searchcited by 58
  19. 2018Deep Neural Networks Learn Non-Smooth Functions Effectivelycited by 57
  20. 2020Generalization to New Actions in Reinforcement Learningcited by 25
  21. 2020GRAC: Self-Guided and Self-Regularized Actor-Criticcited by 21
  22. 2019Q-Learning for Continuous Actions with Cross-Entropy Guided Policiescited by 18
  23. 2023Ensemble Reinforcement Learning: A Surveycited by 16
  24. 2018Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvementcited by 12
  25. 2023Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agentscited by 12
  26. 2021Robotic Surgery With Lean Reinforcement Learningcited by 11
  27. 2017Learning to Run with Actor-Critic Ensemblecited by 8
  28. 2019Quadratic Q-network for Learning Continuous Control for Autonomous Vehiclescited by 8
  29. 2019Learning Probabilistic Multi-Modal Actor Models for Vision-Based Robotic Graspingcited by 5
  30. 2022PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scalecited by 5
  31. 2019Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025When a Robot is More Capable than a Human: Learning from Constrained Demonstratorscited by 1
  2. 2025Actor-Free Continuous Control via Structurally Maximizable Q-Functionscited by 0
  3. 2026Breaking the Grid: Distance-Guided Reinforcement Learning in Large Discrete Action Spacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Compatible Gradient Approximations for Actor-Critic Algorithms
  2. 2025Quasi-Newton Compatible Actor-Critic for Deterministic Policies
  3. 2025Wasserstein Policy Optimization
  4. 2024AFU: Actor-Free critic Updates in off-policy RL for continuous control
  5. 2025Distributions as Actions: A Unified Framework for Diverse Action Spaces
  6. 2026From Policy Evaluation to Off-Policy Control: Provable Emergence of In-Context Q-Learning
  7. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
  8. 2026Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods