Data to 7 Oct 2026

Continuous Deep Q-Learning with Model-based Acceleration

arXiv 1603.00748 ↗First posted 2 Mar 2016cs.LG cs.AI cs.ROcited by 149

Not on the map. Why?

Abstract (arXiv)

Model-free reinforcement learning has been successfully applied to a range of challenging problems, and has recently been extended to handle large neural network policies and value functions. However, the sample complexity of model-free algorithms, particularly when using high-dimensional function approximators, tends to limit their applicability to physical systems. In this paper, we explore algorithms and representations to reduce the sample complexity of deep reinforcement learning for continuous control tasks. We propose two complementary techniques for improving the efficiency of such algorithms. First, we derive a continuous variant of the Q-learning algorithm, which we call normalized adantage functions (NAF), as an alternative to the more commonly used policy gradient and actor-critic methods. NAF representation allows us to apply Q-learning with experience replay to continuous tasks, and substantially improves performance on a set of simulated robotic control tasks. To further improve the efficiency of our approach, we explore the use of learned models for accelerating model-free reinforcement learning. We show that iteratively refitted local linear models are especially effective for this, and demonstrate substantially faster learning on domains where such models are applicable.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 2; 2019: 1; 2020: 0; 2021: 47; 2022: 29; 2023: 20; 2024: 24; 2025: 11; 2026: 14.

149 papers.

Sort:

By area:

2023Mastering Diverse Domains through World Modelscited by 1,193
2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
2022Online Decision Transformercited by 194
2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
2021Learning to drive from a world on railscited by 114
2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
2021Recent Advances in Reinforcement Learning in Financecited by 102
2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
2022q-Learning in Continuous Timecited by 68
2021A Survey of Deep RL and IL for Autonomous Driving Policy Learningcited by 59

38 more

2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
2019Watch, Try, Learn: Meta-Learning from Demonstrations and Rewardcited by 26
2025AnyMAC: Cascading Flexible Multi-Agent Collaboration via Next-Agent Predictioncited by 25
2021Species Distribution Modeling for Machine Learning Practitioners: A Reviewcited by 21
2021Continuous Control with Action Quantization from Demonstrationscited by 21
2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 20
2022Improving aircraft performance using machine learning: a reviewcited by 19
2022A Survey on Reinforcement Learning in Aviation Applicationscited by 19
2022Deep Reinforcement Learning for Inverse Inorganic Materials Designcited by 18
2022On-device Training: A First Overview on Existing Systemscited by 18
2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
2023AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfercited by 17
2023Acquisition Conditioned Oracle for Nongreedy Active Feature Acquisitioncited by 17
2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 17
2021Using Cyber Terrain in Reinforcement Learning for Penetration Testingcited by 16
2024Graph Reinforcement Learning for Combinatorial Optimization: A Survey and Unifying Perspectivecited by 16
2021Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learningcited by 15
2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
2023TD Convergence: An Optimization Perspectivecited by 15
2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 15
2023Holistic Network Virtualization and Pervasive Network Intelligence for 6Gcited by 14
2021High-Accuracy Model-Based Reinforcement Learning, a Surveycited by 13
2024Prioritized Generative Replaycited by 13
2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
2021Braxlines: Fast and Interactive Toolkit for RL-driven Behavior Engineering beyond Reward Maximizationcited by 11
2021Distributed Reinforcement Learning for Privacy-Preserving Dynamic Edge Cachingcited by 10
2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learningcited by 10
2021Applications of the Free Energy Principle to Machine Learning and Neurosciencecited by 9
2021Implicitly Regularized RL with Implicit Q-Valuescited by 9
2021PTR-PPO: Proximal Policy Optimization with Prioritized Trajectory Replaycited by 9
2022Automated Reinforcement Learning: An Overviewcited by 9
2024Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networkscited by 9

50 most cited of 149 All 149 (JSON)