mlsanityData to 7 Oct 2026

Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution

arXiv 2404.04253 ↗First posted 5 Apr 2024cs.LG cs.AI cs.ROcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent reinforcement learning approaches have shown surprisingly strong capabilities of bang-bang policies for solving continuous control benchmarks. The underlying coarse action space discretizations often yield favourable exploration characteristics while final performance does not visibly suffer in the absence of action penalization in line with optimal control theory. In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency, but action costs can be detrimental to exploration during early training. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution, taking advantage of recent results in decoupled Q-learning to scale our approach to high-dimensional action spaces up to dim(A) = 38. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 46 of its 74 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

46 of 74 references.

References · 46

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2020Mastering Atari with Discrete World Modelscited by 990
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  9. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  10. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  11. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  12. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
34 more
  1. 2017Population Based Training of Neural Networkscited by 351
  2. 2018Maximum a Posteriori Policy Optimisationcited by 318
  3. 2018Mean Field Multi-Agent Reinforcement Learningcited by 202
  4. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  5. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  6. 2017Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learningcited by 114
  7. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  8. 2019Deep Coordination Graphscited by 96
  9. 2017Action Branching Architectures for Deep Reinforcement Learningcited by 90
  10. 2021Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharingcited by 85
  11. 2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80
  12. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 65
  13. 2017Hybrid Reward Architecture for Reinforcement Learningcited by 57
  14. 2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 50
  15. 2020Value-Decomposition Multi-Agent Actor-Criticscited by 47
  16. 2020Regularizing Action Policies for Smooth Control with Reinforcement Learningcited by 47
  17. 2018Remember and Forget for Experience Replaycited by 42
  18. 2020Continuous-Discrete Reinforcement Learning for Hybrid Control in Roboticscited by 42
  19. 2021Reinforcement Learning for Adaptive Mesh Refinementcited by 37
  20. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 37
  21. 2019Value constrained model-free continuous controlcited by 34
  22. 2018Relative Entropy Regularized Policy Iterationcited by 32
  23. 2020Q-Learning in enormous action spaces via amortized approximate maximizationcited by 30
  24. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 30
  25. 2019Growing Action Spacescited by 29
  26. 2018Mix&Match - Agent Curricula for Reinforcement Learningcited by 26
  27. 2019Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learningcited by 25
  28. 2020Learning to Represent Action Values as a Hypergraph on the Action Verticescited by 22
  29. 2022Solving Continuous Control via Q-learningcited by 21
  30. 2023Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotioncited by 21
  31. 2017Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learningcited by 15
  32. 2017Parameter Sharing Deep Deterministic Policy Gradient for Cooperative Multi-agent Reinforcement Learningcited by 13
  33. 2019Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learningcited by 13
  34. 2024REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 2.

6 papers.

  1. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  2. 2024Learning in complex action spaces without policy gradientscited by 0
  3. 2025Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Networkcited by 0
  4. 2025Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learningcited by 0
  5. 2026Quantum-Inspired Reinforcement Learning for Secure and Sustainable AIoT-Driven Supply Chain Systemscited by 0
  6. 2026Action-Free Offline-to-Online RL via Discretised State Policiescited by 0