mlsanityData to 7 Oct 2026

Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning

arXiv 2402.03570 ↗First posted 5 Feb 2024cs.LG cs.AIcited by 31

On the map: World models → Latent dynamics models and model-based RL

also in 2 more
Abstract (arXiv)

We introduce Diffusion World Model (DWM), a conditional diffusion model capable of predicting multistep future states and rewards concurrently. As opposed to traditional one-step dynamics models, DWM offers long-horizon predictions in a single forward pass, eliminating the need for recursive queries. We integrate DWM into model-based value estimation, where the short-term return is simulated by future trajectories sampled from DWM. In the context of offline reinforcement learning, DWM can be viewed as a conservative value regularization through generative modeling. Alternatively, it can be seen as a data source that enables offline Q-learning with synthetic data. Our experiments on the D4RL dataset confirm the robustness of DWM to long-horizon simulation. In terms of absolute performance, DWM significantly surpasses one-step dynamics models with a 44% performance gain, and is comparable to or slightly surpassing their model-free counterparts.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 75 of its 90 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

75 of 90 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023World Models via Policy-Guided Trajectory Diffusion13 Dec
  2. 2023TD-MPC2: Scalable, Robust World Models for Continuous Control25 Oct
  3. 2023Transformer-based World Models Are Happy With 100k Interactions13 Mar
  4. 2023Mastering Diverse Domains through World Models10 Jan
  5. 2022Transformers are Sample-Efficient World Models1 Sep
  6. 2022TransDreamer: Reinforcement Learning with Transformer World Models19 Feb
  7. 2021Mastering Atari Games with Limited Data30 Oct
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  9. 2020Mastering Atari with Discrete World Models5 Oct
  10. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  11. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun
  12. 2018World Models27 Mar
1 more
  1. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning8 Aug

Other references · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2022Flow Matching for Generative Modelingcited by 5,426
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2023Consistency Modelscited by 1,778
38 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2018Group Normalizationcited by 1,514
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2021Variational Diffusion Modelscited by 1,145
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  13. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  14. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  15. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  16. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  17. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  18. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  19. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  20. 2019Model-Based Reinforcement Learning for Ataricited by 495
  21. 2023Learning Interactive Real-World Simulatorscited by 440
  22. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  23. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  24. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  25. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  26. 2022Multi-Game Decision Transformerscited by 212
  27. 2022Online Decision Transformercited by 194
  28. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  29. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  30. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  31. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 151
  32. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  33. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  34. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 105
  35. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  36. 2023Synthetic Experience Replaycited by 100
  37. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  38. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 12; 2026: 8.

31 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  3. 2024Diffusion Policy Policy Optimizationcited by 255
  4. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  5. 2025Flow Q-Learningcited by 162
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  7. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  8. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  9. 2025Pre-Trained Video Generative Models as World Simulatorscited by 43
  10. 2024Diffusion Model Predictive Controlcited by 35
  11. 2024DiffuSolve: Diffusion-based Solver for Non-convex Trajectory Optimizationcited by 19
  12. 2024Diffusion Spectral Representation for Reinforcement Learningcited by 15
19 more
  1. 2024Prioritized Generative Replaycited by 13
  2. 2025Implicit Search via Discrete Diffusion: A Study on Chesscited by 11
  3. 2025DMWM: Dual-Mind World Model with Long-Term Imaginationcited by 10
  4. 2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Modelscited by 8
  5. 2024Diffusion-Reinforcement Learning Hierarchical Motion Planning in Multi-agent Adversarial Gamescited by 7
  6. 2025Adaptive Inference-Time Scaling via Cyclic Diffusion Searchcited by 7
  7. 2025Intention-Conditioned Flow Occupancy Modelscited by 7
  8. 2025Unleashing Flow Policies with Distributional Criticscited by 7
  9. 2024Solving Continual Offline RL through Selective Weights Activation on Aligned Spacescited by 6
  10. 2025Exploratory Diffusion Model for Unsupervised Reinforcement Learningcited by 4
  11. 2026World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applicationscited by 3
  12. 2026JEDI: Joint Embedding Diffusion World Model for Online Model-Based Reinforcement Learningcited by 2
  13. 2026Reversal Q-Learningcited by 2
  14. 2026Improving Zero-Shot Offline RL via Behavioral Task Samplingcited by 1
  15. 2026Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RLcited by 1
  16. 2024Diffusion-based subsurface CO2 multiphysics monitoring and forecastingcited by 0
  17. 2025Performance Asymmetry in Model-Based Reinforcement Learningcited by 0
  18. 2026Reinforcement Learning: From Algorithms To Foundation Modelscited by 0
  19. 2026Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions
  2. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
  3. 2024Off-dynamics Conditional Diffusion Planners
  4. 2025Diffusion Policies with Value-Conditional Optimization for Offline Reinforcement Learning
  5. 2026Diffusion Modulation via Environment Mechanism Modeling for Planning
  6. 2024Diffusion Models as Optimizers for Efficient Planning in Offline RL
  7. 2026Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization
  8. 2024DyDiff: Long-Horizon Rollout via Dynamics Diffusion for Offline Reinforcement Learning