mlsanityData to 7 Oct 2026

Learning Dynamics Models for Model Predictive Agents

arXiv 2109.14311 ↗First posted 29 Sep 2021cs.LG cs.ROcited by 21

Not on the map. Why?

Abstract (arXiv)

Model-Based Reinforcement Learning involves learning a textitdynamics model from data, and then using this model to optimise behaviour, most often with an online textitplanner. Much of the recent research along these lines presents a particular set of design choices, involving problem definition, model learning and planning. Given the multiple contributions, it is difficult to evaluate the effects of each. This paper sets out to disambiguate the role of different design choices for learning dynamics models, by comparing their performance to planning with a ground-truth model -- the simulator. First, we collect a rich dataset from the training sequence of a model-free agent on 5 domains of the DeepMind Control Suite. Second, we train feed-forward dynamics models in a supervised fashion, and evaluate planner performance while varying and analysing different model design choices, including ensembling, stochasticity, multi-step training and timestep size. Besides the quantitative analysis, we describe a set of qualitative findings, rules of thumb, and future research directions for planning with learned dynamics models. Videos of the results are available at https://sites.google.com/view/learning-better-models.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 29 Sep 2021.

Builds on

39 of 55 references.

References · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2018World Modelscited by 1,444
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2017Hindsight Experience Replaycited by 1,096
  10. 2020Mastering Atari with Discrete World Modelscited by 990
  11. 2018DeepMind Control Suitecited by 935
  12. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
27 more
  1. 2020Learning Mesh-Based Simulation with Graph Networkscited by 840
  2. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  4. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  5. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  6. 2018Maximum a Posteriori Policy Optimisationcited by 318
  7. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  8. 2018Graph networks as learnable physics engines for inference and controlcited by 242
  9. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  10. 2019Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learningcited by 205
  11. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  12. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  13. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  14. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  15. 2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 90
  16. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 87
  17. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  18. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 46
  19. 2019Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learningcited by 40
  20. 2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 35
  21. 2019Deep Lagrangian Networks for end-to-end learning of energy-based control for under-actuated systemscited by 32
  22. 2020Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloadscited by 27
  23. 2020Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learningcited by 27
  24. 2019Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Modelscited by 26
  25. 2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 25
  26. 2020Learning Accurate Long-term Dynamics for Model-based Reinforcement Learningcited by 21
  27. 2020A Differentiable Newton Euler Algorithm for Multi-body Model Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 7; 2024: 5; 2025: 2; 2026: 1.

21 papers.

Sort:

By area:

  1. 2022Physical Design using Differentiable Learned Simulatorscited by 37
  2. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 31
  3. 2023A Generalist Dynamics Model for Controlcited by 24
  4. 2021Combining Physics and Deep Learning to learn Continuous-Time Dynamics Modelscited by 20
  5. 2023Simplified Temporal Consistency Reinforcement Learningcited by 17
  6. 2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
  7. 2022On Uncertainty in Deep State Space Models for Model-Based Reinforcement Learningcited by 13
  8. 2024Locality Sensitive Sparse Encoding for Learning World Models Onlinecited by 12
  9. 2022Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processescited by 10
  10. 2023Neural Laplace Control for Continuous-time Delayed Systemscited by 10
  11. 2024iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learningcited by 9
  12. 2025Discrete Codebook World Models for Continuous Controlcited by 9
9 more
  1. 2023Bridging the Gap Between Multi-Step and One-Shot Trajectory Prediction via Self-Supervisioncited by 7
  2. 2024Adaptive Regularization of Representation Rank as an Implicit Constraint of Bellman Equationcited by 7
  3. 2023Investigating the role of model-based learning in exploration and transfercited by 5
  4. 2023Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learningcited by 5
  5. 2023Get Back Here: Robust Imitation by Return-to-Distribution Planningcited by 4
  6. 2025Time-Aware World Model for Adaptive Prediction and Controlcited by 4
  7. 2024A Multi-step Loss Function for Robust Learning of the Dynamics in Model-based Reinforcement Learningcited by 1
  8. 2024Diminishing Return of Value Expansion Methodscited by 0
  9. 2026Denoising Particle Filters: Learning State Estimation with Single-Step Objectivescited by 0