mlsanityData to 7 Oct 2026

Solving Continuous Control via Q-learning

arXiv 2210.12566 ↗First posted 22 Oct 2022cs.LG cs.AI cs.ROcited by 21

Not on the map. Why?

Abstract (arXiv)

While there has been substantial success for solving continuous control with actor-critic methods, simpler critic-only methods such as Q-learning find limited application in the associated high-dimensional action spaces. However, most actor-critic methods come at the cost of added complexity: heuristics for stabilisation, compute requirements and wider hyperparameter search spaces. We show that a simple modification of deep Q-learning largely alleviates these issues. By combining bang-bang action discretization with value decomposition, framing single-agent control as cooperative multi-agent reinforcement learning (MARL), this simple critic-only approach matches performance of state-of-the-art continuous actor-critic methods when learning from features or pixels. We extend classical bandit examples from cooperative MARL to provide intuition for how decoupled critics leverage state information to coordinate joint optimization, and demonstrate surprisingly strong performance across a variety of continuous control tasks.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 53 of its 76 references that are in the map. First posted on arXiv 22 Oct 2022.

Builds on

53 of 76 references.

References · 53

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  7. 2018World Modelscited by 1,444
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2015Prioritized Experience Replaycited by 1,070
  12. 2020Mastering Atari with Discrete World Modelscited by 990
38 more
  1. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  2. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  5. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  6. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  7. 2016Deep Exploration via Bootstrapped DQNcited by 518
  8. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  9. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  10. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  11. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  12. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  13. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  14. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  15. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  16. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  17. 2018Maximum a Posteriori Policy Optimisationcited by 318
  18. 2018Mean Field Multi-Agent Reinforcement Learningcited by 202
  19. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  20. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  21. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  22. 2017Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learningcited by 114
  23. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  24. 2019Deep Coordination Graphscited by 96
  25. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
  26. 2017Action Branching Architectures for Deep Reinforcement Learningcited by 90
  27. 2021Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharingcited by 85
  28. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 65
  29. 2017Hybrid Reward Architecture for Reinforcement Learningcited by 57
  30. 2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 50
  31. 2020Value-Decomposition Multi-Agent Actor-Criticscited by 47
  32. 2020Continuous-Discrete Reinforcement Learning for Hybrid Control in Roboticscited by 42
  33. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 37
  34. 2020Q-Learning in enormous action spaces via amortized approximate maximizationcited by 30
  35. 2019Growing Action Spacescited by 29
  36. 2020Learning to Represent Action Values as a Hypergraph on the Action Verticescited by 22
  37. 2017Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learningcited by 15
  38. 2020Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensemblescited by 14

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 6; 2026: 2.

21 papers.

Sort:

By area:

  1. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 34
  2. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  3. 2024Green Screen Augmentation Enables Scene Generalisation in Robotic Manipulationcited by 11
  4. 2024Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learningcited by 10
  5. 2023No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learningcited by 6
  6. 2024REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processescited by 6
  7. 2024Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolutioncited by 6
  8. 2024Stochastic Q-learning for Large Discrete Action Spacescited by 5
  9. 2024Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learningcited by 4
  10. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spacescited by 3
  11. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 3
  12. 2026Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Controlcited by 2
9 more
  1. 2023A Statistical Guarantee for Representation Transfer in Multitask Imitation Learningcited by 1
  2. 2024How to discretize continuous state-action spaces in Q-learning: A symbolic control approachcited by 0
  3. 2024Learning in complex action spaces without policy gradientscited by 0
  4. 2025Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Networkcited by 0
  5. 2025Q-function Decomposition with Intervention Semantics with Factored Action Spacescited by 0
  6. 2025Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learningcited by 0
  7. 2025Distributions as Actions: A Unified Framework for Diverse Action Spacescited by 0
  8. 2025Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functionscited by 0
  9. 2026Action-Free Offline-to-Online RL via Discretised State Policiescited by 0