mlsanityData to 7 Oct 2026

TD-MPC2: Scalable, Robust World Models for Continuous Control

arXiv 2310.16828 ↗First posted 25 Oct 2023cs.LG cs.AI cs.CVcited by 482

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

TD-MPC is a model-based reinforcement learning (RL) algorithm that performs local trajectory optimization in the latent space of a learned implicit (decoder-free) world model. In this work, we present TD-MPC2: a series of improvements upon the TD-MPC algorithm. We demonstrate that TD-MPC2 improves significantly over baselines across 104 online RL tasks spanning 4 diverse task domains, achieving consistently strong results with a single set of hyperparameters. We further show that agent capabilities increase with model and data size, and successfully train a single 317M parameter agent to perform 80 tasks across multiple task domains, embodiments, and action spaces. We conclude with an account of lessons, opportunities, and risks associated with large TD-MPC2 agents. Explore videos, models, data, code, and more at https://tdmpc2.com

Key facts. Cited by 482 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 51 of its 66 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

51 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Finetuning Offline World Models in the Real World24 Oct
  2. 2023A Generalist Dynamics Model for Control18 May
  3. 2023Mastering Diverse Domains through World Models10 Jan
  4. 2021Mastering Atari Games with Limited Data30 Oct

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2016Layer Normalizationcited by 5,579
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2015Continuous control with deep reinforcement learningcited by 3,279
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  12. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
35 more
  1. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  2. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  5. 2022A Generalist Agentcited by 802
  6. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  8. 2017Deep Reinforcement Learning that Matterscited by 727
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  11. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  12. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  13. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  14. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  15. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  16. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  17. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  18. 2022Multi-Game Decision Transformerscited by 212
  19. 2019Mish: A Self Regularized Non-Monotonic Activation Functioncited by 141
  20. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  21. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  22. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 87
  23. 2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80
  24. 2021Learning and Planning in Complex Action Spacescited by 76
  25. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  26. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  27. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
  28. 2023MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulationcited by 27
  29. 2022Simplicial Embeddings in Self-Supervised Learning and Downstream Classificationcited by 23
  30. 2022Improving Policy Optimization with Generalist-Specialist Learningcited by 19
  31. 2023RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictabilitycited by 17
  32. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  33. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 15
  34. 2023IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Controlcited by 14
  35. 2022EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Modelcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 69; 2025: 151; 2026: 260.

482 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  3. 2026Causal World Modeling for Robot Controlcited by 289
  4. 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 274
  5. 2024Navigation World Modelscited by 262
  6. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelscited by 237
  7. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  8. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  9. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  10. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  11. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  12. 2025Training Agents Inside of Scalable World Modelscited by 122
38 more
  1. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  2. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  3. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  4. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  5. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101
  6. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 79
  7. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
  8. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  9. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  10. 2025World-Env: Leveraging World Model as a Virtual Environment for VLA Post-Trainingcited by 57
  11. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 56
  12. 2025Horizon Reduction Makes RL Scalablecited by 55
  13. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  14. 2025Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Roboticscited by 47
  15. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  16. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  17. 2026Learning Latent Action World Models In The Wildcited by 42
  18. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  19. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 41
  20. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  21. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  22. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 39
  23. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 39
  24. 2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 38
  25. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 38
  26. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
  27. 2025What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?cited by 37
  28. 2024Diffusion Model Predictive Controlcited by 35
  29. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  30. 2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulationcited by 35
  31. 2026Do World Action Models Generalize Better than VLAs? A Robustness Studycited by 35
  32. 2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 34
  33. 2026Hierarchical Planning with Latent World Modelscited by 34
  34. 2025Flow-Based Policy for Online Reinforcement Learningcited by 33
  35. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 32
  36. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  37. 2026Temporal Straightening for Latent Planningcited by 32
  38. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31

50 most cited of 482 All 482 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
  2. 2026Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models
  3. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimization
  4. 2026Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution
  5. 2026Dual-Axis Policy Optimization for LLM Agents: Bayesian Feedback Attribution and Trajectory Mass Normalization
  6. 2026WAM-RL: World-Action Model Reinforcement Learning with Reconstruction Rewards and Online Video SFT
  7. 2025Continual Reinforcement Learning by Planning with Online World Models
  8. 2025Controllable Flow Matching for Online Reinforcement Learning