mlsanityData to 7 Oct 2026

Temporal Difference Learning for Model Predictive Control

arXiv 2203.04955 ↗First posted 9 Mar 2022cs.LG cs.ROcited by 375

On the map: World models → Latent predictive world models

Abstract (arXiv)

Data-driven model predictive control has two key advantages over model-free methods: a potential for improved sample efficiency through model learning, and better performance as computational budget for planning increases. However, it is both costly to plan over long horizons and challenging to obtain an accurate model of the environment. In this work, we combine the strengths of model-free and model-based methods. We use a learned task-oriented latent dynamics model for local trajectory optimization over a short horizon, and use a learned terminal value function to estimate long-term return, both of which are learned jointly by temporal difference learning. Our method, TD-MPC, achieves superior sample efficiency and asymptotic performance over prior work on both state and image-based continuous control tasks from DMControl and Meta-World. Code and video results are available at https://nicklashansen.github.io/td-mpc.

Key facts. Cited by 375 papers in the map (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 49 of its 69 references that are in the map. First posted on arXiv 9 Mar 2022.

Builds on

49 of 69 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Learning Latent Dynamics for Planning from Pixels12 Nov

Other references · 48

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2020Mastering Atari with Discrete World Modelscited by 990
  12. 2018DeepMind Control Suitecited by 935
36 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  6. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  8. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2019Model-Based Reinforcement Learning for Ataricited by 495
  11. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  12. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  13. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  14. 2021Mastering Atari Games with Limited Datacited by 235
  15. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  16. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  17. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  18. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  19. 2020Model-Based Offline Planningcited by 126
  20. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  21. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  22. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  23. 2019Exploring Model-based Planning with Policy Networkscited by 90
  24. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  25. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
  26. 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 54
  27. 2018CEM-RL: Combining evolutionary and gradient-based methods for policy searchcited by 53
  28. 2020Learning Off-Policy with Online Planningcited by 46
  29. 2021Learning to Jump from Pixelscited by 39
  30. 2020Blending MPC & Value Function Approximation for Efficient Reinforcement Learningcited by 27
  31. 2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 25
  32. 2020GRAC: Self-Guided and Self-Regularized Actor-Criticcited by 20
  33. 2019Information Theoretic Model Predictive Q-Learningcited by 16
  34. 2018SOLAR: Deep Structured Representations for Model-Based Reinforcement Learningcited by 14
  35. 2021The Value of Planning for Infinite-Horizon Model Predictive Controlcited by 8
  36. 2019Curious Meta-Controller: Adaptive Alternation between Model-Based and Model-Free Control in Deep Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 18; 2023: 46; 2024: 71; 2025: 114; 2026: 126.

375 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  3. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  4. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  5. 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 274
  6. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelscited by 237
  7. 2025Real-Time Execution of Action Chunking Flow Policiescited by 199
  8. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  9. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  10. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  11. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  12. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 79
38 more
  1. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
  2. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  3. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  4. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
  5. 2023TorchRL: A data-driven decision-making library for PyTorchcited by 59
  6. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  7. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 55
  8. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  9. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  10. 2022Skill-based Model-based Reinforcement Learningcited by 50
  11. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  12. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  13. 2025Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Roboticscited by 47
  14. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  15. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  16. 2025Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysiscited by 44
  17. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  18. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  19. 2024CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agentscited by 41
  20. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  21. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
  22. 2024DiffuserLite: Towards Real-time Diffusion Planningcited by 40
  23. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  24. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  25. 2023Finetuning Offline World Models in the Real Worldcited by 39
  26. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Datacited by 39
  27. 2024Autoregressive Models in Vision: A Surveycited by 39
  28. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 39
  29. 2025DynaGuide: Steering Diffusion Polices with Active Dynamic Guidancecited by 39
  30. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 39
  31. 2026VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Modelcited by 39
  32. 2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 38
  33. 2023Model-Based Runtime Monitoring with Interactive Imitation Learningcited by 37
  34. 2024Diffusion Model Predictive Controlcited by 35
  35. 2026Do World Action Models Generalize Better than VLAs? A Robustness Studycited by 35
  36. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  37. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  38. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32

50 most cited of 375 All 375 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2018Differentiable MPC for End-to-end Planning and Control
  2. 2024Faster Model Predictive Control via Self-Supervised Initialization Learning
  3. 2022Learning to Optimize in Model Predictive Control
  4. 2023Contrastive Example-Based Control
  5. 2026Directed Temporal Representations for Offline Visual Control
  6. 2026World Model Control by Trajectory Reachability Metrics
  7. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning
  8. 2026Beyond a single latent space: a dual-latent world model for long-horizon planning