mlsanityData to 7 Oct 2026

TesserAct: Learning 4D Embodied World Models

arXiv 2504.20995 ↗First posted 29 Apr 2025cs.CV cs.ROcited by 90

On the map: World models → Embodied and robot world models

Abstract (arXiv)

This paper presents an effective approach for learning novel 4D embodied world models, which predict the dynamic evolution of 3D scenes over time in response to an embodied agent's actions, providing both spatial and temporal consistency. We propose to learn a 4D world model by training on RGB-DN (RGB, Depth, and Normal) videos. This not only surpasses traditional 2D models by incorporating detailed shape, configuration, and temporal changes into their predictions, but also allows us to effectively learn accurate inverse dynamic models for an embodied agent. Specifically, we first extend existing robotic manipulation video datasets with depth and normal information leveraging off-the-shelf models. Next, we fine-tune a video generation model on this annotated dataset, which jointly predicts RGB-DN (RGB, Depth, and Normal) for each frame. We then present an algorithm to directly convert generated RGB, Depth, and Normal videos into a high-quality 4D scene of the world. Our method ensures temporal and spatial coherence in 4D scene predictions from embodied scenarios, enables novel view synthesis for embodied environments, and facilitates policy learning that significantly outperforms those derived from prior video-based world models.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 60 of its 78 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

60 of 78 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Aether: Geometric-Aware Unified World Modeling24 Mar
  2. 2024IRASim: A Fine-Grained World Model for Robot Manipulation20 Jun
  3. 2024RoboDreamer: Learning Compositional World Models for Robot Imagination18 Apr
  4. 20243D-VLA: A 3D Vision-Language-Action Generative World Model14 Mar
  5. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  11. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  12. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
38 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  4. 2020Mastering Atari with Discrete World Modelscited by 990
  5. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  6. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  7. 2024Genie: Generative Interactive Environmentscited by 706
  8. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  9. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  10. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  11. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  12. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  13. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  14. 2023Learning Interactive Real-World Simulatorscited by 440
  15. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  16. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  17. 2022Transformers are Sample-Efficient World Modelscited by 317
  18. 2024Shape of Motion: 4D Reconstruction from a Single Videocited by 247
  19. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  20. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  21. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 212
  22. 2023Text-To-4D Dynamic Scene Generationcited by 204
  23. 2022ECON: Explicit Clothed humans Optimized via Normal integrationcited by 199
  24. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
  25. 2023DreamGaussian4D: Generative 4D Gaussian Splattingcited by 191
  26. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  27. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  28. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  29. 2024StableNormal: Reducing Diffusion Variance for Stable and Sharp Normalcited by 153
  30. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  31. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistencycited by 143
  32. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  33. 2018State Representation Learning for Control: An Overviewcited by 125
  34. 2023Consistent4D: Consistent 360deg Dynamic Object Generation from Monocular Videocited by 124
  35. 2024Rethinking Inductive Biases for Surface Normal Estimationcited by 122
  36. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistencycited by 117
  37. 2023Animate124: Animating One Image to 4D Dynamic Scenecited by 113
  38. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 84

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 34; 2026: 56.

90 papers.

Sort:

By area:

  1. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 86
  2. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  3. 2026WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Modelscited by 68
  4. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 67
  5. 2025Large Video Planner Enables Generalizable Robot Controlcited by 67
  6. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  7. 2026Unified 4D World Action Modeling from Video Priors with Asynchronous Denoisingcited by 65
  8. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 60
  9. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  10. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  11. 2025World-in-World: World Models in a Closed-Loop Worldcited by 44
  12. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
38 more
  1. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  2. 2025RoboScape: Physics-informed Embodied World Modelcited by 38
  3. 2025VRAG: Learning World Models for Interactive Video Generationcited by 37
  4. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 36
  5. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 33
  6. 2026ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?cited by 33
  7. 20254DNeX: Feed-Forward 4D Generative Modeling Made Easycited by 31
  8. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  9. 2026Rethinking Video Generation Model for the Embodied Worldcited by 30
  10. 2025Learning 3D Persistent Embodied World Modelscited by 29
  11. 2026Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectoriescited by 24
  12. 2025Geometry-aware 4D Video Generation for Robot Manipulationcited by 23
  13. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 23
  14. 2025WristWorld: Generating Wrist-Views via 4D World Models for Robotic Manipulationcited by 21
  15. 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Modelscited by 19
  16. 2025Robot Learning from a Physical World Modelcited by 18
  17. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 16
  18. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  19. 2025Vidarc: Embodied Video Diffusion Model for Closed-loop Controlcited by 15
  20. 20254D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Timecited by 13
  21. 2026MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulationcited by 13
  22. 2026WAM4D: Fast 4D World Action Model via Spatial Register Tokenscited by 12
  23. 2026Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generationcited by 12
  24. 2025Mask2IV: Interaction-Centric Video Generation via Mask Trajectoriescited by 10
  25. 2026Action Images: End-to-End Policy Learning via Multiview Video Generationcited by 10
  26. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
  27. 2026World Action Models: A Surveycited by 10
  28. 2026RoboStereo: Dual-Tower 4D Embodied World Models for Unified Policy Optimizationcited by 9
  29. 2026Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulationcited by 9
  30. 20264D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fieldscited by 9
  31. 2025RoboTransfer: Controllable Geometry-Consistent Video Diffusion for Manipulation Policy Transfercited by 7
  32. 2025LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoEcited by 7
  33. 2026ReWorld: Multi-Dimensional Reward Modeling for Embodied World Modelscited by 7
  34. 2026CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoningcited by 7
  35. 2026GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulationcited by 7
  36. 2026PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulationcited by 7
  37. 2026Masked Visual Actions for Unified World Modelingcited by 7
  38. 2025RoDyn: Taming Interactive Robot-Dynamic 2.5D World Model for Robotic Manipulationcited by 6

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation
  2. 2026Embody4D: A Generalist Data Engine for Embodied 4D World Modeling
  3. 2025Any4D: Open-Prompt 4D Generation from Natural Language and Images
  4. 2026DepthWorld: 3D World Model for Robot Manipulation
  5. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation
  6. 2026XGenAct: Geometry-Enhanced World Action Models through Cross-Task Generation
  7. 2026Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directions
  8. 2026SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy