mlsanityData to 7 Oct 2026

Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

arXiv 1708.02596 ↗First posted 8 Aug 2017cs.LG cs.AI cs.ROcited by 288

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Model-free deep reinforcement learning algorithms have been shown to be capable of learning a wide range of robotic skills, but typically require a very large number of samples to achieve good performance. Model-based algorithms, in principle, can provide for much more efficient learning, but have proven difficult to extend to expressive, high-capacity models such as deep neural networks. In this work, we demonstrate that medium-sized neural network models can in fact be combined with model predictive control (MPC) to achieve excellent sample complexity in a model-based reinforcement learning algorithm, producing stable and plausible gaits to accomplish various complex locomotion tasks. We also propose using deep neural network dynamics models to initialize a model-free learner, in order to combine the sample efficiency of model-based approaches with the high task-specific performance of model-free methods. We empirically demonstrate on MuJoCo locomotion tasks that our pure model-based approach trained on just random action data can follow arbitrary trajectories with excellent sample efficiency, and that our hybrid algorithm can accelerate model-free learning on high-speed benchmark tasks, achieving sample efficiency gains of 3-5x on swimmer, cheetah, hopper, and ant agents. Videos can be found at https://sites.google.com/view/mbmf

Key facts. Cited by 288 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 0 of its 46 references that are in the map. First posted on arXiv 8 Aug 2017.

Builds on

0 of 46 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 6; 2019: 6; 2020: 11; 2021: 78; 2022: 44; 2023: 44; 2024: 43; 2025: 31; 2026: 25.

288 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2018World Modelscited by 1,444
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  5. 2020Mastering Atari with Discrete World Modelscited by 990
  6. 2021E(n) Equivariant Graph Neural Networkscited by 951
  7. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  12. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
38 more
  1. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  4. 2023Structured World Models from Human Videoscited by 144
  5. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  6. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  7. 2019Exploring Model-based Planning with Policy Networkscited by 90
  8. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  9. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  10. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  11. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  12. 2022A Survey on Model-based Reinforcement Learningcited by 65
  13. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 64
  14. 2025Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Roboticscited by 47
  15. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  16. 2021Learning Neuro-Symbolic Relational Transition Models for Bilevel Planningcited by 43
  17. 2023A Survey on Causal Reinforcement Learningcited by 43
  18. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimationcited by 42
  19. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learningcited by 42
  20. 2021Continuous-Time Model-Based Reinforcement Learningcited by 39
  21. 2023Adaptive Online Replanning with Diffusion Modelscited by 33
  22. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  23. 2021Few-shot Language Coordination by Modeling Theory of Mindcited by 32
  24. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  25. 2023Rethinking Closed-loop Training for Autonomous Drivingcited by 29
  26. 2024ASID: Active Exploration for System Identification in Robotic Manipulationcited by 27
  27. 2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 25
  28. 2021Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rolloutscited by 24
  29. 2021Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Criticcited by 23
  30. 2021Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Controlcited by 22
  31. 2023Explainable Reinforcement Learning via a Causal World Modelcited by 22
  32. 2021Learning Dynamics Models for Model Predictive Agentscited by 21
  33. 2021Safe Control with Neural Network Dynamic Modelscited by 21
  34. 2021Where to go next: Learning a Subgoal Recommendation Policy for Navigation Among Pedestrianscited by 20
  35. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
  36. 2022A Survey on Reinforcement Learning in Aviation Applicationscited by 19
  37. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 19
  38. 2021VisuoSpatial Foresight for Physical Sequential Fabric Manipulationcited by 18

50 most cited of 288 All 288 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024An Efficient Model-Based Approach on Learning Agile Motor Skills without Reinforcement
  2. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning
  3. 2026Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL
  4. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations
  5. 2025Flexible Locomotion Learning with Diffusion Model Predictive Control
  6. 2023DecAP: Decaying Action Priors for Accelerated Imitation Learning of Torque-Based Legged Locomotion Policies
  7. 2023Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion
  8. 2020Neural Dynamic Policies for End-to-End Sensorimotor Learning