mlsanityData to 14 Oct 2026

Learning in complex action spaces without policy gradients

arXiv 2410.06317 ↗First posted 8 Oct 2024cs.LG cs.AI stat.MLcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

While conventional wisdom holds that policy gradient methods are better suited to complex action spaces than action-value methods, foundational work has shown that the two paradigms are equivalent in small, finite action spaces (O'Donoghue et al., 2017; Schulman et al., 2017a). This raises the question of why their computational applicability and performance diverge as the complexity of the action space increases. We hypothesize that the apparent superiority of policy gradients in such settings stems not from intrinsic qualities of the paradigm but from universal principles that can also be applied to action-value methods, enabling similar functions. We identify three such principles and provide a framework for incorporating them into action-value methods. To support our hypothesis, we instantiate this framework in what we term QMLE, for Q-learning with maximum likelihood estimation. Our results show that QMLE can be applied to complex action spaces at a computational cost comparable to that of policy gradient methods, all without using policy gradients. Furthermore, QMLE exhibits strong performance on the DeepMind Control Suite, even when compared to state-of-the-art methods such as DMPO and D4PG.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 54 of its 84 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

54 of 84 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution5 Apr

Other references · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2016Layer Normalizationcited by 5,669
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  10. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  11. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  12. 2015Prioritized Experience Replaycited by 1,099
38 more
  1. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  2. 2018DeepMind Control Suitecited by 956
  3. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  4. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  7. 2019Solving Rubik's Cube with a Robot Handcited by 734
  8. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 575
  9. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  10. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  11. 2018Maximum a Posteriori Policy Optimisationcited by 328
  12. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  13. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  14. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  15. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 161
  16. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
  17. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  18. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  19. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  20. 2022On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networkscited by 120
  21. 2017Action Branching Architectures for Deep Reinforcement Learningcited by 92
  22. 2017Time Limits in Reinforcement Learningcited by 79
  23. 2021Learning and Planning in Complex Action Spacescited by 77
  24. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 68
  25. 2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 51
  26. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 38
  27. 2020Q-Learning in enormous action spaces via amortized approximate maximizationcited by 30
  28. 2019Growing Action Spacescited by 29
  29. 2018Expected Policy Gradients for Reinforcement Learningcited by 28
  30. 2022Human-level Atari 200x fastercited by 28
  31. 2022Solving Continuous Control via Q-learningcited by 24
  32. 2020Learning to Represent Action Values as a Hypergraph on the Action Verticescited by 23
  33. 2019Distributional Policy Optimization: An Alternative Approach for Continuous Controlcited by 22
  34. 2020Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarkingcited by 19
  35. 2017Expected Policy Gradientscited by 16
  36. 2019Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learningcited by 16
  37. 2017Mean Actor Criticcited by 8
  38. 2019All-Action Policy Gradient Methods: A Numerical Integration Approachcited by 4

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare
  2. 2024Mixed Q-Functionals: Advancing Value-Based Methods in Cooperative MARL with Continuous Action Domains
  3. 2026Revisiting Overestimation Bias Problem of Q-learning: Settling Large Discrete Action Space via Action Intersection
  4. 2024Learning Agents With Prioritization and Parameter Noise in Continuous State and Action Space
  5. 2026Selective Growing Q-Networks (SGQN): An Analysis of Adaptive Action Grids in Decoupled Q-Learning
  6. 2024Stochastic Q-learning for Large Discrete Action Spaces
  7. 2022Learning Value Functions from Undirected State-only Experience
  8. 2025Actor-Free Continuous Control via Structurally Maximizable Q-Functions