# Robot-Factored World Models via Robot Rendering arXiv: https://arxiv.org/abs/2607.22535 ยท HTML: https://mlsanity.com/paper/2607.22535/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in Embodied and robot world models, one of 11 directions in World models. - Builds on 62 of its 68 references that are in the map. - First posted on arXiv 24 Jul 2026. ## Placed in - World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/) ## Builds on (most cited first, 30 of 62) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022) - [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022) - [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025) - [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025) - [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023) - [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023) - [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018) - [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023) - [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025) - [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025) - [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024) - [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018) - [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024) - [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023) - [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023) - [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025) - [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023) - [UniVLA: Learning to Act Anywhere with Task-centric Latent Actions](https://mlsanity.com/paper/2505.06111/) (2025) - [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024) - [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022) - [ControlVideo: Training-free Controllable Text-to-Video Generation](https://mlsanity.com/paper/2305.13077/) (2023) - [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021) - [Diffusion for World Modeling: Visual Details Matter in Atari](https://mlsanity.com/paper/2405.12399/) (2024) - [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023) - [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024) - [Navigation World Models](https://mlsanity.com/paper/2412.03572/) (2024) - [DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory](https://mlsanity.com/paper/2308.08089/) (2023) ## Built on by (most cited first, 3 of 3) - [DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation](https://mlsanity.com/paper/2608.13489/) (2026) - [Hydra-0: Action Flow for Generalist World Modeling and Control](https://mlsanity.com/paper/2608.18077/) (2026) - [RoGSW4RLD: Feed-Forward 4D Gaussian Lifting for Robot World Model Rollouts](https://mlsanity.com/paper/2609.35311/) (2026) ## Abstract (arXiv) Action-conditioned video world models predict future observations from an initial observation and an action signal. In robotics, actions influence future observations through two distinct processes: they are first realized into robot motion by the robot body and controller, and the scene then responds through contact and object motion. Conditioning directly on action commands asks the world model to learn the realization process itself, while conditioning on logged future states leaks the interaction outcomes it is meant to predict. We propose robot-factored world models, which move two robot-specific factors outside the world model. First, action realization: each command is rolled through the robot's own controller and kinematics into a deployment-available nominal trajectory, a middle signal that avoids both action-realization learning and future-state leakage. Second, robot rendering: this nominal trajectory is rendered through the robot URDF, factoring the robot's geometry, kinematics, and appearance out of the model and into explicit rendered robot geometry. To resolve depth ambiguity, we pair end-effector depth with scene depth, giving geometric cues for contact and occlusion beyond image-plane overlap. Together, camera-aware static RGB/depth context and rendered robot geometry form a shared visual world-model interface that stays consistent across viewpoints and robot embodiments, so the model sees the action only as visible robot geometry and learns how objects respond to it. Our experiments show that the rendered interface outperforms vector-conditioned baselines and generalizes to unseen robot embodiments at inference. We further demonstrate that our model generates robot manipulation videos from human demonstrations by retargeting and rendering the hand motion as robot geometry.