mlsanityData to 14 Oct 2026

Whole-Body Conditioned Egocentric Video Prediction

arXiv 2506.21552 ↗First posted 26 Jun 2025cs.CV cs.AI cs.LGcited by 30

Not on the map. Why?

Cited by 22 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We train models to Predict Ego-centric Video from human Actions (PEVA), given the past video and an action represented by the relative 3D body pose. By conditioning on kinematic pose trajectories, structured by the joint hierarchy of the body, our model learns to simulate how physical human actions shape the environment from a first-person point of view. We train an auto-regressive conditional diffusion transformer on Nymeria, a large-scale dataset of real-world egocentric video and body pose capture. We further design a hierarchical evaluation protocol with increasingly challenging tasks, enabling a comprehensive analysis of the model's embodied prediction and control abilities. Our work represents an initial attempt to tackle the challenges of modeling complex real-world environments and embodied agent behaviors with video prediction from the perspective of a human.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 58 of its 78 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

58 of 78 references.

References · 58

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  10. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  11. 2018World Modelscited by 1,513
  12. 2023Mastering Diverse Domains through World Modelscited by 1,253
38 more
  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  2. 2022Human Motion Diffusion Modelcited by 1,080
  3. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  4. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  5. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  6. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  7. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
  8. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 688
  9. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  10. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
  11. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  12. 2023Learning Interactive Real-World Simulatorscited by 458
  13. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  14. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 386
  15. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 368
  16. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 337
  17. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  18. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 324
  19. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  20. 2024Navigation World Modelscited by 272
  21. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 248
  22. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 230
  23. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  24. 2021Anticipative Video Transformercited by 200
  25. 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 189
  26. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 158
  27. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 154
  28. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 142
  29. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 139
  30. 2024Expressive Whole-Body Control for Humanoid Robotscited by 139
  31. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 130
  32. 2020Rolling-Unrolling LSTMs for Action Anticipation from First-Person Videocited by 120
  33. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 114
  34. 20213D Neural Scene Representations for Visuomotor Controlcited by 111
  35. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 94
  36. 2019Ego-Pose Estimation and Forecasting as Real-Time PD Controlcited by 87
  37. 2024Video as the New Language for Real-World Decision Makingcited by 87
  38. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 23.

30 papers.

Sort:

By area:

  1. 2025Training Agents Inside of Scalable World Modelscited by 136
  2. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  3. 2025Cambrian-S: Towards Spatial Supersensing in Videocited by 134
  4. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  5. 2026Learning Latent Action World Models In The Wildcited by 47
  6. 2026Hierarchical Planning with Latent World Modelscited by 43
  7. 2026Temporal Straightening for Latent Planningcited by 38
  8. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 31
  9. 2026LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Modelscited by 28
  10. 2026BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Maskscited by 20
  11. 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Datacited by 20
  12. 2025Dexterous World Modelscited by 18
18 more
  1. 2026LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Modelcited by 13
  2. 2026Egocentric World Model for Photorealistic Hand-Object Interaction Synthesiscited by 12
  3. 2025EgoControl: Controllable Egocentric Video Generation via 3D Full-Body Posescited by 8
  4. 2025GrndCtrl: Grounding World Models via Self-Supervised Reward Alignmentcited by 4
  5. 2025TV2TV: A Unified Framework for Interleaved Language and Video Generationcited by 3
  6. 2026Grounding Generated Videos in Feasible Plans via World Modelscited by 3
  7. 2026NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Datacited by 3
  8. 2026EgoExo-WM: Unlocking Exo Video for Ego World Modelscited by 3
  9. 2026AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customizationcited by 3
  10. 2026Lifting Embodied World Models for Planning and Controlcited by 2
  11. 2026E3C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Controlcited by 2
  12. 2026Unifying Object-Centric World Models and Diffusion Policy: A Hierarchical Framework for Multi-Stage Robotic Taskscited by 2
  13. 2026HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Controlcited by 2
  14. 2026The Seriality Gap in Video Diffusion Modelscited by 2
  15. 2026How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Predictioncited by 1
  16. 2026Not All Actions Are Equal: Rethinking Conditioning for Dexterous World Modelcited by 1
  17. 2026Real-Time Human-Centric World Modeling for Upper-Body Human-Object Interactioncited by 1
  18. 2026Ego-Exo4D Human Meshes Dataset: 4D Human Motion Reconstruction for Ego-Exo Capturescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Walk through Paintings: Egocentric World Models from Internet Priors
  2. 2026EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE
  3. 2024Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data
  4. 2024EgoMimic: Scaling Imitation Learning via Egocentric Video
  5. 2026ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining
  6. 2025EgoAgent: A Joint Predictive Agent Model in Egocentric Worlds
  7. 2025Ego-centric Predictive Model Conditioned on Hand Trajectories
  8. 2026EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR