mlsanityData to 14 Oct 2026

EgoTwin: Dreaming Body and View in First Person

arXiv 2508.13013 ↗First posted 18 Aug 2025cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

While exocentric video synthesis has achieved great progress, egocentric video generation remains largely underexplored, which requires modeling first-person view content along with camera motion patterns induced by the wearer's body movements. To bridge this gap, we introduce a novel task of joint egocentric video and human motion generation, characterized by two key challenges: 1) Viewpoint Alignment: the camera trajectory in the generated video must accurately align with the head trajectory derived from human motion; 2) Causal Interplay: the synthesized human motion must causally align with the observed visual dynamics across adjacent video frames. To address these challenges, we propose EgoTwin, a joint video-motion generation framework built on the diffusion transformer architecture. Specifically, EgoTwin introduces a head-centric motion representation that anchors the human motion to the head joint and incorporates a cybernetics-inspired interaction mechanism that explicitly captures the causal interplay between video and motion within attention operations. For comprehensive evaluation, we curate a large-scale real-world dataset of synchronized text-video-motion triplets and design novel metrics to assess video-motion consistency. Extensive experiments demonstrate the effectiveness of the EgoTwin framework.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 57 of its 64 references that are in the map. First posted on arXiv 18 Aug 2025.

Builds on

57 of 64 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
38 more
  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  2. 2022Video Diffusion Modelscited by 2,305
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  5. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  6. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  8. 2022Human Motion Diffusion Modelcited by 1,080
  9. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  10. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
  11. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  13. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 687
  14. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  15. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 588
  16. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  17. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
  18. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  19. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
  20. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
  21. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  22. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  23. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 441
  24. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 400
  25. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 359
  26. 2016The KIT Motion-Language Datasetcited by 338
  27. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 332
  28. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  29. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 239
  30. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 227
  31. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 208
  32. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  33. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  34. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 187
  35. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 158
  36. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 154
  37. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  38. 2023MMM: Generative Masked Motion Modelcited by 131

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2025UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generation
  2. 2026EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generation
  3. 2024EgoGen: An Egocentric Synthetic Data Generator
  4. 2024Intention-driven Ego-to-Exo Video Generation
  5. 2025EgoControl: Controllable Egocentric Video Generation via 3D Full-Body Poses
  6. 2025EgoX: Egocentric Video Generation from a Single Exocentric Video
  7. 2025WorldWander: Bridging Egocentric and Exocentric Worlds in Video Generation
  8. 2026Beyond Visual Quality: Evaluating Physical Consistency under Ego-Motion with EgoGenEval