# EgoSim: Egocentric World Simulator for Embodied Interaction Generation

arXiv: https://arxiv.org/abs/2604.01001 · HTML: https://mlsanity.com/paper/2604.01001/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 50 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Builds on (most cited first, 30 of 40)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details](https://mlsanity.com/paper/2507.02546/) (2025)
- [EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video](https://mlsanity.com/paper/2505.11709/) (2025)
- [Matrix-game 2.0: An open-source real-time and streaming interactive world model](https://mlsanity.com/paper/2508.13009/) (2025)
- [ViPE: Video Pose Engine for 3D Geometric Perception](https://mlsanity.com/paper/2508.10934/) (2025)
- [Advancing Open-source World Models](https://mlsanity.com/paper/2601.20540/) (2026)
- [Scaling Up Dynamic Human-Scene Interaction Modeling](https://mlsanity.com/paper/2403.08629/) (2024)
- [Recent Advances in 3D Gaussian Splatting](https://mlsanity.com/paper/2403.11134/) (2024)
- [Yume-1.5: A Text-Controlled Interactive World Generation Model](https://mlsanity.com/paper/2512.22096/) (2025)
- [The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control](https://mlsanity.com/paper/2412.03568/) (2024)
- [GeoCalib: Learning Single-image Calibration with Geometric Optimization](https://mlsanity.com/paper/2409.06704/) (2024)
- [EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation](https://mlsanity.com/paper/2411.08380/) (2024)
- [Spatia: Video Generation with Updatable Spatial Memory](https://mlsanity.com/paper/2512.15716/) (2025)
- [LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Models](https://mlsanity.com/paper/2603.07145/) (2026)
- [In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Data](https://mlsanity.com/paper/2511.15704/) (2025)
- [WristWorld: Generating Wrist-Views via 4D World Models for Robotic Manipulation](https://mlsanity.com/paper/2510.07313/) (2025)
- [PlayerOne: Egocentric World Simulator](https://mlsanity.com/paper/2506.09995/) (2025)

## Built on by (most cited first, 14 of 14)

- [AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization](https://mlsanity.com/paper/2606.07326/) (2026)
- [RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation](https://mlsanity.com/paper/2607.06558/) (2026)
- [HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Control](https://mlsanity.com/paper/2607.02075/) (2026)
- [EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE](https://mlsanity.com/paper/2607.28243/) (2026)
- [EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation](https://mlsanity.com/paper/2605.18214/) (2026)
- [Aero-World: Action-Conditioned Aerial Video Generation from Inertial Controls](https://mlsanity.com/paper/2605.19728/) (2026)
- [PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph](https://mlsanity.com/paper/2608.01905/) (2026)
- [Human-Centric Intelligence in the Era of Foundation Models: A Survey](https://mlsanity.com/paper/2608.18184/) (2026)
- [Skel-WAM: A Hand-Skeleton-Conditioned World Action Model for Human-to-Robot Manipulation Transfer](https://mlsanity.com/paper/2609.21514/) (2026)
- [Code Plans, Diffusion Renders: Open-Ended Generative World Modeling](https://mlsanity.com/paper/2609.26458/) (2026)
- [Learning What to Recall: Adaptive Multi-Cue Episodic Memory for World Models](https://mlsanity.com/paper/2609.34677/) (2026)
- [Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction](https://mlsanity.com/paper/2610.12299/) (2026)
- [Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics](https://mlsanity.com/paper/2610.13522/) (2026)
- [Generative World Simulation with DAO and JING](https://mlsanity.com/paper/2610.19726/) (2026)

## Abstract (arXiv)

We introduce EgoSim, a closed-loop egocentric world simulator that generates spatially consistent interaction videos and persistently updates the underlying 3D scene state for continuous simulation. Existing egocentric simulators either lack explicit 3D grounding, causing structural drift under viewpoint changes, or treat the scene as static, failing to update world states across multi-stage interactions. EgoSim addresses both limitations by modeling 3D scenes as updatable world states. We generate embodiment interactions via a Geometry-action-aware Observation Simulation model, with spatial consistency from an Interaction-aware State Updating module. To overcome the critical data bottleneck posed by the difficulty in acquiring densely aligned scene-interaction training pairs, we design a scalable pipeline that extracts static point clouds, camera trajectories, and embodiment actions from in-the-wild large-scale monocular egocentric videos. We further introduce EgoCap, a capture system that enables low-cost real-world data collection with uncalibrated smartphones. Extensive experiments demonstrate that EgoSim significantly outperforms existing methods in terms of visual quality, spatial consistency, and generalization to complex scenes and in-the-wild dexterous interactions, while supporting cross-embodiment transfer to robotic manipulation. Codes and datasets will be open soon. The project page is at egosimulator.github.io.