mlsanityData to 14 Oct 2026

FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation

arXiv 2505.10075 ↗First posted 15 May 2025cs.RO cs.CVcited by 17

On the map: World models → Embodied and robot world models

Abstract (arXiv)

This paper investigates training better visual world models for robot manipulation, i.e., models that can predict future visual observations by conditioning on past frames and robot actions. Specifically, we consider world models that operate on RGB-D frames (RGB-D world models). As opposed to canonical approaches that handle dynamics prediction mostly implicitly and reconcile it with visual rendering in a single model, we introduce FlowDreamer, which adopts 3D scene flow as explicit motion representations. FlowDreamer first predicts 3D scene flow from past frame and action conditions with a U-Net, and then a diffusion model will predict the future frame utilizing the scene flow. FlowDreamer is trained end-to-end despite its modularized nature. We conduct experiments on 4 different benchmarks, covering both video prediction and visual planning tasks. The results demonstrate that FlowDreamer achieves better performance compared to other baseline RGB-D world models by 7% on semantic similarity, 11% on pixel quality, and 6% on success rate in various robot manipulation domains.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 82 of its 100 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

82 of 100 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation3 Jan
  2. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Models24 May
  3. 2024RoboDreamer: Learning Compositional World Models for Robot Imagination18 Apr
  4. 20243D-VLA: A 3D Vision-Language-Action Generative World Model14 Mar
  5. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models16 Oct

Other references · 77

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  10. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
38 more
  1. 2021Vision Transformers for Dense Predictioncited by 1,803
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  3. 2024Depth Anything V2cited by 1,422
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  5. 2023Mastering Diverse Domains through World Modelscited by 1,253
  6. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  7. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  8. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  9. 2020Mastering Atari with Discrete World Modelscited by 1,007
  10. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  11. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  12. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  13. 2024Genie: Generative Interactive Environmentscited by 747
  14. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  15. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  16. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  17. 2019Model-Based Reinforcement Learning for Ataricited by 502
  18. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  19. 2023Learning Interactive Real-World Simulatorscited by 458
  20. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 424
  21. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  22. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  23. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  24. 2022Transformers are Sample-Efficient World Modelscited by 328
  25. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 309
  26. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  27. 2024Diffusion Models Are Real-Time Game Enginescited by 296
  28. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  29. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  30. 2022FlowFormer: A Transformer Architecture for Optical Flowcited by 275
  31. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  32. 2018FlowNet3D: Learning Scene Flow in 3D Point Cloudscited by 242
  33. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 230
  34. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  35. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 211
  36. 2025Video Depth Anything: Consistent Depth Estimation for Super-Long Videoscited by 207
  37. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 181
  38. 2024SpatialTracker: Tracking Any 2D Pixels in 3D Spacecited by 181

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 12.

17 papers.

Sort:

By area:

  1. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  2. 2026Unified 4D World Action Modeling from Video Priors with Asynchronous Denoisingcited by 71
  3. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  4. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 51
  5. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  6. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 25
  7. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  8. 2026Action Images: End-to-End Policy Learning via Multiview Video Generationcited by 12
  9. 2026World Action Models: A Surveycited by 12
  10. 2026World2Act: Latent Action Post-Training from World Model Dynamicscited by 8
  11. 2026DDP-WM: Disentangled Dynamics Prediction for Efficient World Modelscited by 5
  12. 2026From World Models to World Action Models: A Concise Tutorial for Roboticscited by 4
5 more
  1. 2026The Embodiment Gap in Robot Foundation Modelscited by 1
  2. 2025Envision: Embodied Visual Planning via Goal-Imagery Video Diffusioncited by 0
  3. 2026LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulationcited by 0
  4. 2026Sensitivity Shaping for Latent Modelingcited by 0
  5. 2026Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation
  2. 2026FlowVLA: Enabling World Dynamics Understanding through Pluggable 3D Flow Learning
  3. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model
  4. 2026KineWorld: Action-Induced Transport Fields for Embodied World Modeling
  5. 2026PACT-WAM: Predicting Actions and Visual Foresight with Compact Temporal Encoding for Robot Manipulation
  6. 2026NarrativeFlow: Flow-Based Vision-Language-Action Model Using Robot Velocity Fields
  7. 2026SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy
  8. 2026MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation