mlsanityData to 7 Oct 2026

Walk through Paintings: Egocentric World Models from Internet Priors

arXiv 2601.15284 ↗First posted 21 Jan 2026cs.CVcited by 13

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

What if a video generation model could not only imagine a plausible future, but the correct one -- accurately reflecting how the world changes with each action? We answer this by presenting the Egocentric World Model (EgoWM), a simple, architecture-agnostic method that transforms any pre-trained video diffusion model into an action-conditioned world model, enabling precisely controllable future prediction. Rather than training from scratch, we repurpose the rich world priors of Internet-scale video models by injecting appropriately compressed motor commands through lightweight conditioning layers. This allows our model to follow actions faithfully while preserving generalization and realism. Our approach scales naturally across embodiments and action spaces -- from 3-DoF mobile robots to 25-DoF humanoids, where predicting egocentric joint-angle-driven dynamics is substantially more challenging. The model produces coherent rollouts for both navigation and manipulation, requiring only modest fine-tuning. To evaluate physical correctness independent of appearance, we introduce the Structural Consistency Score (SCS), which measures whether stable scene elements evolve consistently with the provided actions. Our method improves SCS by up to 65% over the prior state of the art, Navigation World Models; applies seamlessly to three different video diffusion model architectures; and effectively utilizes Internet priors to generalize to unseen environments, including navigation and manipulation inside paintings. Finally, we demonstrate the applicability of EgoWM to robotic planning.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 48 of its 64 references that are in the map. First posted on arXiv 21 Jan 2026.

Builds on

48 of 64 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulation11 Oct
  2. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets3 Apr
  3. 2024IRASim: A Fine-Grained World Model for Robot Manipulation20 Jun
  4. 2022Masked World Models for Visual Control28 Jun

Other references · 44

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  7. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  9. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  11. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  12. 2018World Modelscited by 1,444
32 more
  1. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  2. 2025SAM 3: Segment Anything with Conceptscited by 835
  3. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  4. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  5. 2023ModelScope Text-to-Video Technical Reportcited by 651
  6. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  7. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  8. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  9. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  10. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  11. 2024Diffusion Models Are Real-Time Game Enginescited by 282
  12. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  13. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  14. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  15. 2024Navigation World Modelscited by 262
  16. 2017Stochastic Variational Video Predictioncited by 234
  17. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  18. 2025Video models are zero-shot learners and reasonerscited by 208
  19. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  20. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  21. 2022Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigationcited by 122
  22. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  23. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 116
  24. 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
  25. 2025Video Generators are Robot Policiescited by 86
  26. 2024pix2gestalt: Amodal Segmentation by Synthesizing Wholescited by 70
  27. 2025AllTracker: Efficient Dense Point Tracking at High Resolutioncited by 54
  28. 2022TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Modelscited by 48
  29. 2024Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentationcited by 27
  30. 2024ReferEverything: Towards Segmenting Everything We Can Speak of in Videoscited by 10
  31. 2025Benchmarking Image Similarity Metrics for Novel View Synthesis Applicationscited by 5
  32. 2025GrndCtrl: Grounding World Models via Self-Supervised Reward Alignmentcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data
  2. 2026EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE
  3. 2026EgoForge: Goal-Directed Egocentric World Simulator
  4. 2026EgoExo-WM: Unlocking Exo Video for Ego World Models
  5. 2026Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
  6. 2026E3C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Control
  7. 2026Foresight Without Seeing: Latent Futures for World Action Models
  8. 2026AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization