EgoTwin: Dreaming Body and View in First Person
Not on the map. Why?
Abstract (arXiv)
While exocentric video synthesis has achieved great progress, egocentric video generation remains largely underexplored, which requires modeling first-person view content along with camera motion patterns induced by the wearer's body movements. To bridge this gap, we introduce a novel task of joint egocentric video and human motion generation, characterized by two key challenges: 1) Viewpoint Alignment: the camera trajectory in the generated video must accurately align with the head trajectory derived from human motion; 2) Causal Interplay: the synthesized human motion must causally align with the observed visual dynamics across adjacent video frames. To address these challenges, we propose EgoTwin, a joint video-motion generation framework built on the diffusion transformer architecture. Specifically, EgoTwin introduces a head-centric motion representation that anchors the human motion to the head joint and incorporates a cybernetics-inspired interaction mechanism that explicitly captures the causal interplay between video and motion within attention operations. For comprehensive evaluation, we curate a large-scale real-world dataset of synchronized text-video-motion triplets and design novel metrics to assess video-motion consistency. Extensive experiments demonstrate the effectiveness of the EgoTwin framework.
Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 57 of its 64 references that are in the map. First posted on arXiv 18 Aug 2025.
Builds on
57 of 64 references.
References · 57
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2022Classifier-Free Diffusion Guidancecited by 5,847
- 2017Neural Discrete Representation Learningcited by 4,712
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
38 more
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
- 2022Video Diffusion Modelscited by 2,305
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
- 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
- 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
- 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
- 2022Human Motion Diffusion Modelcited by 1,080
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
- 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
- 2022Autoregressive Image Generation using Residual Quantizationcited by 722
- 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
- 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 687
- 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
- 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 588
- 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
- 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
- 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
- 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
- 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
- 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
- 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
- 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 441
- 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 400
- 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 359
- 2016The KIT Motion-Language Datasetcited by 338
- 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 332
- 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
- 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 239
- 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 227
- 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 208
- 2021Stochastic Scene-Aware Motion Predictioncited by 203
- 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
- 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 187
- 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 158
- 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 154
- 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
- 2023MMM: Generative Masked Motion Modelcited by 131
50 most cited of 57 All 57 (JSON)
Built on by
Citing papers per year: 2025: 4; 2026: 9.
13 papers.
Sort:
By area:
- 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
- 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
- 2026EgoForge: Goal-Directed Egocentric World Simulatorcited by 12
- 2025UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generationcited by 7
- 2025VA-π: Variational Policy Alignment for Pixel-Aware Autoregressive Generationcited by 7
- 2025EgoReAct: Egocentric Video-Driven 3D Human Reaction Generationcited by 7
- 2026Unleashing Guidance Without Classifiers for Human-Object Interaction Animationcited by 6
- 2026NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Datacited by 3
- 2026What if? Emulative Simulation with World Models for Situated Reasoningcited by 2
- 2026E3C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Controlcited by 2
- 2026RelaxFlow: Text-Driven Amodal 3D Generationcited by 1
- 2026EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generationcited by 0
1 more
- 2026Being-M0.7: A Latent World-Action Model for Humanoid Robots8 Oct 2026
- 2026E3C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Control25 May 2026
- 2026EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generation21 Apr 2026
- 2026Unleashing Guidance Without Classifiers for Human-Object Interaction Animation26 Mar 2026
- 2026EgoForge: Goal-Directed Egocentric World Simulator20 Mar 2026
- 2026NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data19 Mar 2026
- 2026What if? Emulative Simulation with World Models for Situated Reasoning6 Mar 2026
- 2026RelaxFlow: Text-Driven Amodal 3D Generation5 Mar 2026
- 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos6 Feb 2026
- 2025EgoReAct: Egocentric Video-Driven 3D Human Reaction Generation28 Dec 2025
- 2025AstraNav-World: World Model for Foresight Control and Consistency25 Dec 2025
- 2025VA-π: Variational Policy Alignment for Pixel-Aware Autoregressive Generation22 Dec 2025
- 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
- 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
- 2026EgoForge: Goal-Directed Egocentric World Simulatorcited by 12
- 2026What if? Emulative Simulation with World Models for Situated Reasoningcited by 2
- 2026Being-M0.7: A Latent World-Action Model for Humanoid Robotscited by 0
- 2026Being-M0.7: A Latent World-Action Model for Humanoid Robots8 Oct 2026
- 2026EgoForge: Goal-Directed Egocentric World Simulator20 Mar 2026
- 2026What if? Emulative Simulation with World Models for Situated Reasoning6 Mar 2026
- 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos6 Feb 2026
- 2025AstraNav-World: World Model for Foresight Control and Consistency25 Dec 2025
- 2025UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generationcited by 7
- 2025EgoReAct: Egocentric Video-Driven 3D Human Reaction Generationcited by 7
- 2026Unleashing Guidance Without Classifiers for Human-Object Interaction Animationcited by 6
- 2026NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Datacited by 3
- 2026E3C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Controlcited by 2
- 2026RelaxFlow: Text-Driven Amodal 3D Generationcited by 1
- 2026E3C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Control25 May 2026
- 2026Unleashing Guidance Without Classifiers for Human-Object Interaction Animation26 Mar 2026
- 2026NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data19 Mar 2026
- 2026RelaxFlow: Text-Driven Amodal 3D Generation5 Mar 2026
- 2025EgoReAct: Egocentric Video-Driven 3D Human Reaction Generation28 Dec 2025
- 2025UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation8 Dec 2025
Similar, not cited
Close in wording, no citation either way
- 2025UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generation
- 2026EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generation
- 2024EgoGen: An Egocentric Synthetic Data Generator
- 2024Intention-driven Ego-to-Exo Video Generation
- 2025EgoControl: Controllable Egocentric Video Generation via 3D Full-Body Poses
- 2025EgoX: Egocentric Video Generation from a Single Exocentric Video
- 2025WorldWander: Bridging Egocentric and Exocentric Worlds in Video Generation
- 2026Beyond Visual Quality: Evaluating Physical Consistency under Ego-Motion with EgoGenEval