Walk through Paintings: Egocentric World Models from Internet Priors
On the map: World models → Embodied and robot world models
also in 1 more
Abstract (arXiv)
What if a video generation model could not only imagine a plausible future, but the correct one -- accurately reflecting how the world changes with each action? We answer this by presenting the Egocentric World Model (EgoWM), a simple, architecture-agnostic method that transforms any pre-trained video diffusion model into an action-conditioned world model, enabling precisely controllable future prediction. Rather than training from scratch, we repurpose the rich world priors of Internet-scale video models by injecting appropriately compressed motor commands through lightweight conditioning layers. This allows our model to follow actions faithfully while preserving generalization and realism. Our approach scales naturally across embodiments and action spaces -- from 3-DoF mobile robots to 25-DoF humanoids, where predicting egocentric joint-angle-driven dynamics is substantially more challenging. The model produces coherent rollouts for both navigation and manipulation, requiring only modest fine-tuning. To evaluate physical correctness independent of appearance, we introduce the Structural Consistency Score (SCS), which measures whether stable scene elements evolve consistently with the provided actions. Our method improves SCS by up to 65% over the prior state of the art, Navigation World Models; applies seamlessly to three different video diffusion model architectures; and effectively utilizes Internet priors to generalize to unseen environments, including navigation and manipulation inside paintings. Finally, we demonstrate the applicability of EgoWM to robotic planning.
Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 48 of its 64 references that are in the map. First posted on arXiv 21 Jan 2026.
Builds on
48 of 64 references.
Closest ancestors · 4
Same direction, newest first
- 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulation11 Oct
- 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets3 Apr
- 2024IRASim: A Fine-Grained World Model for Robot Manipulation20 Jun
- 2022Masked World Models for Visual Control28 Jun
Other references · 44
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
- 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
- 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
- 2018World Modelscited by 1,444
32 more
- 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
- 2025SAM 3: Segment Anything with Conceptscited by 835
- 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
- 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
- 2023ModelScope Text-to-Video Technical Reportcited by 651
- 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
- 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
- 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
- 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
- 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
- 2024Diffusion Models Are Real-Time Game Enginescited by 282
- 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
- 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
- 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
- 2024Navigation World Modelscited by 262
- 2017Stochastic Variational Video Predictioncited by 234
- 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
- 2025Video models are zero-shot learners and reasonerscited by 208
- 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
- 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
- 2022Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigationcited by 122
- 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
- 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 116
- 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
- 2025Video Generators are Robot Policiescited by 86
- 2024pix2gestalt: Amodal Segmentation by Synthesizing Wholescited by 70
- 2025AllTracker: Efficient Dense Point Tracking at High Resolutioncited by 54
- 2022TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Modelscited by 48
- 2024Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentationcited by 27
- 2024ReferEverything: Towards Segmenting Everything We Can Speak of in Videoscited by 10
- 2025Benchmarking Image Similarity Metrics for Novel View Synthesis Applicationscited by 5
- 2025GrndCtrl: Grounding World Models via Self-Supervised Reward Alignmentcited by 4
Built on by
Citing papers per year: 2026: 13.
13 papers.
Sort:
By area:
- 2026WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RLcited by 46
- 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
- 2026ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Modelscited by 9
- 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 6
- 2026Human Cognition in Machines: A Unified Perspective of World Modelscited by 5
- 2026Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Jointscited by 3
- 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoningcited by 3
- 2026How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Predictioncited by 1
- 2026Drift-Resistant Navigation World Model with Anchored Epipolar Guidancecited by 1
- 2026World Models for Embodied Intelligence: From Plausible to Controllable to Actionablecited by 1
- 2026EgoPlay: Event-Triggered Video Editing for Egocentric Streamscited by 0
- 2026Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Modelscited by 0
- 2026World Models for Embodied Intelligence: From Plausible to Controllable to Actionable15 Sep 2026
- 2026Beyond Visual Quality: Evaluating Physical Consistency under Ego-Motion with EgoGenEval10 Sep 2026
- 2026Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Models3 Sep 2026
- 2026EgoPlay: Event-Triggered Video Editing for Egocentric Streams27 Jul 2026
- 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends31 May 2026
- 2026Drift-Resistant Navigation World Model with Anchored Epipolar Guidance23 May 2026
- 2026How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Prediction19 May 2026
- 2026ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models9 May 2026
- 2026World Model for Robot Learning: A Comprehensive Survey30 Apr 2026
- 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning29 Apr 2026
- 2026Human Cognition in Machines: A Unified Perspective of World Models17 Apr 2026
- 2026Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints12 Mar 2026
1 more
- 2026WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RLcited by 46
- 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
- 2026ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Modelscited by 9
- 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 6
- 2026Human Cognition in Machines: A Unified Perspective of World Modelscited by 5
- 2026Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Jointscited by 3
- 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoningcited by 3
- 2026Drift-Resistant Navigation World Model with Anchored Epipolar Guidancecited by 1
- 2026World Models for Embodied Intelligence: From Plausible to Controllable to Actionablecited by 1
- 2026Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Modelscited by 0
- 2026World Models for Embodied Intelligence: From Plausible to Controllable to Actionable15 Sep 2026
- 2026Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Models3 Sep 2026
- 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends31 May 2026
- 2026Drift-Resistant Navigation World Model with Anchored Epipolar Guidance23 May 2026
- 2026ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models9 May 2026
- 2026World Model for Robot Learning: A Comprehensive Survey30 Apr 2026
- 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning29 Apr 2026
- 2026Human Cognition in Machines: A Unified Perspective of World Models17 Apr 2026
- 2026Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints12 Mar 2026
- 2026WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL15 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data
- 2026EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE
- 2026EgoForge: Goal-Directed Egocentric World Simulator
- 2026EgoExo-WM: Unlocking Exo Video for Ego World Models
- 2026Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
- 2026E3C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Control
- 2026Foresight Without Seeing: Latent Futures for World Action Models
- 2026AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization