# Robot Learning from a Physical World Model

arXiv: https://arxiv.org/abs/2511.07416 · HTML: https://mlsanity.com/paper/2511.07416/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 67 of its 71 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 30 of 67)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos](https://mlsanity.com/paper/2412.04463/) (2024)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [Any-point Trajectory Modeling for Policy Learning](https://mlsanity.com/paper/2401.00025/) (2023)
- [AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains](https://mlsanity.com/paper/2212.08333/) (2022)
- [Unified Video Action Model](https://mlsanity.com/paper/2503.00200/) (2025)
- [Learning to Act from Actionless Videos through Dense Correspondences](https://mlsanity.com/paper/2310.08576/) (2023)
- [Video Language Planning](https://mlsanity.com/paper/2310.10625/) (2023)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)
- [Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation](https://mlsanity.com/paper/2412.15109/) (2024)
- [Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2409.16283/) (2024)
- [BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects](https://mlsanity.com/paper/2303.14158/) (2023)
- [Structured World Models from Human Videos](https://mlsanity.com/paper/2308.10901/) (2023)
- [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024)
- [Flow as the Cross-Domain Manipulation Interface](https://mlsanity.com/paper/2407.15208/) (2024)
- [Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2405.01527/) (2024)
- [AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos](https://mlsanity.com/paper/1912.04443/) (2019)
- [TesserAct: Learning 4D Embodied World Models](https://mlsanity.com/paper/2504.20995/) (2025)
- [Automated Creation of Digital Cousins for Robust Policy Learning](https://mlsanity.com/paper/2410.07408/) (2024)
- [Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos](https://mlsanity.com/paper/2101.07241/) (2021)

## Built on by (most cited first, 20 of 20)

- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [World Model for Robot Learning: A Comprehensive Survey](https://mlsanity.com/paper/2605.00080/) (2026)
- [MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation](https://mlsanity.com/paper/2602.09878/) (2026)
- [SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation](https://mlsanity.com/paper/2602.16863/) (2026)
- [World Action Models: A Survey](https://mlsanity.com/paper/2606.20781/) (2026)
- [Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models](https://mlsanity.com/paper/2603.16065/) (2026)
- [WorldPack: Dynamic Frame Compression for Long-context Video World Modeling](https://mlsanity.com/paper/2512.02473/) (2025)
- [RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models](https://mlsanity.com/paper/2605.03821/) (2026)
- [FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation](https://mlsanity.com/paper/2602.13444/) (2026)
- [DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models](https://mlsanity.com/paper/2603.16860/) (2026)
- [Robot Learning from Human Videos: A Survey](https://mlsanity.com/paper/2604.27621/) (2026)
- [Vesta: A Generalist Embodied Reasoning Model](https://mlsanity.com/paper/2606.20905/) (2026)
- [CRAFT: Video Diffusion for Bimanual Robot Data Generation](https://mlsanity.com/paper/2604.03552/) (2026)
- [OpenWorldLib: A Unified Codebase and Definition of Advanced World Models](https://mlsanity.com/paper/2604.04707/) (2026)
- [ChronoAgentic: A Code-based Multi-Agent World Simulator for Physically Grounded Simulation Construction](https://mlsanity.com/paper/2605.14398/) (2026)
- [3D Generation for Embodied AI and Robotic Simulation: A Survey](https://mlsanity.com/paper/2604.26509/) (2026)
- [World Models for Embodied Intelligence: From Plausible to Controllable to Actionable](https://mlsanity.com/paper/2609.16697/) (2026)
- [Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Models](https://mlsanity.com/paper/2609.03927/) (2026)
- [PhysFlow: Physics-Aware Optical Flow for Motion Controllable Video Generation](https://mlsanity.com/paper/2609.08215/) (2026)
- [CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization](https://mlsanity.com/paper/2610.12567/) (2026)

## Abstract (arXiv)

We introduce PhysWorld, a framework that enables robot learning from video generation through physical world modeling. Recent video generation models can synthesize photorealistic visual demonstrations from language commands and images, offering a powerful yet underexplored source of training signals for robotics. However, directly retargeting pixel motions from generated videos to robots neglects physics, often resulting in inaccurate manipulations. PhysWorld addresses this limitation by coupling video generation with physical world reconstruction. Given a single image and a task command, our method generates task-conditioned videos and reconstructs the underlying physical world from the videos, and the generated video motions are grounded into physically accurate actions through object-centric residual reinforcement learning with the physical world model. This synergy transforms implicit visual guidance into physically executable robotic trajectories, eliminating the need for real robot data collection and enabling zero-shot generalizable robotic manipulation. Experiments on diverse real-world tasks demonstrate that PhysWorld substantially improves manipulation accuracy compared to previous approaches. Visit hrefhttps://pointscoder.github.io/PhysWorldWeb/the project webpage for details.