# Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution

arXiv: https://arxiv.org/abs/2404.04253 · HTML: https://mlsanity.com/paper/2404.04253/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 46 of its 74 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 30 of 46)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [FACMAC: Factored Multi-Agent Centralised Policy Gradients](https://mlsanity.com/paper/2003.06709/) (2020)
- [Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1702.08887/) (2017)
- [Deep Coordination Graphs](https://mlsanity.com/paper/1910.00091/) (2019)
- [Off-Policy Multi-Agent Decomposed Policy Gradients](https://mlsanity.com/paper/2007.12322/) (2020)
- [Action Branching Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1711.08946/) (2017)
- [Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing](https://mlsanity.com/paper/2102.07475/) (2021)
- [MyoSuite -- A contact-rich simulation suite for musculoskeletal motor control](https://mlsanity.com/paper/2205.13600/) (2022)
- [Discretizing Continuous Action Space for On-Policy Optimization](https://mlsanity.com/paper/1901.10500/) (2019)
- [Regularizing Action Policies for Smooth Control with Reinforcement Learning](https://mlsanity.com/paper/2012.06644/) (2020)
- [Hybrid Reward Architecture for Reinforcement Learning](https://mlsanity.com/paper/1706.04208/) (2017)
- [Discrete Sequential Prediction of Continuous Actions for Deep RL](https://mlsanity.com/paper/1705.05035/) (2017)
- [Value-Decomposition Multi-Agent Actor-Critics](https://mlsanity.com/paper/2007.12306/) (2020)
- [Continuous-Discrete Reinforcement Learning for Hybrid Control in Robotics](https://mlsanity.com/paper/2001.00449/) (2020)
- [Remember and Forget for Experience Replay](https://mlsanity.com/paper/1807.05827/) (2018)

## Built on by (most cited first, 7 of 7)

- [Continuous Control with Coarse-to-fine Reinforcement Learning](https://mlsanity.com/paper/2407.07787/) (2024)
- [Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning](https://mlsanity.com/paper/2506.05968/) (2025)
- [Learning in complex action spaces without policy gradients](https://mlsanity.com/paper/2410.06317/) (2024)
- [Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network](https://mlsanity.com/paper/2502.00288/) (2025)
- [Quantum-Inspired Reinforcement Learning for Secure and Sustainable AIoT-Driven Supply Chain Systems](https://mlsanity.com/paper/2601.22339/) (2026)
- [Action-Free Offline-to-Online RL via Discretised State Policies](https://mlsanity.com/paper/2602.00629/) (2026)
- [Selective Growing Q-Networks (SGQN): An Analysis of Adaptive Action Grids in Decoupled Q-Learning](https://mlsanity.com/paper/2610.20463/) (2026)

## Abstract (arXiv)

Recent reinforcement learning approaches have shown surprisingly strong capabilities of bang-bang policies for solving continuous control benchmarks. The underlying coarse action space discretizations often yield favourable exploration characteristics while final performance does not visibly suffer in the absence of action penalization in line with optimal control theory. In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency, but action costs can be detrimental to exploration during early training. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution, taking advantage of recent results in decoupled Q-learning to scale our approach to high-dimensional action spaces up to dim(A) = 38. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.