mlsanityData to 14 Oct 2026

Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videos

arXiv 2512.16907 ↗First posted 18 Dec 2025cs.CV cs.AI cs.ROcited by 8

Not on the map. Why?

Abstract (arXiv)

Prior works on 3D hand trajectory prediction are constrained by datasets that decouple motion from semantic supervision and by models that weakly link reasoning and action. To address these, we first present the EgoMAN dataset, a large-scale egocentric dataset for interaction stage-aware 3D hand trajectory prediction with 219K 6DoF trajectories and 3M structured QA pairs for semantic, spatial, and motion reasoning. We then introduce the EgoMAN model, a reasoning-to-motion framework that links vision-language reasoning and motion generation via a trajectory-token interface. Trained progressively to align reasoning with motion dynamics, our approach yields accurate and stage-aware trajectories with generalization across real-world scenes.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 50 of its 60 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

50 of 60 references.

References · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  10. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
38 more
  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  2. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  4. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  5. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  6. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  7. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  8. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 395
  9. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 359
  10. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  11. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  12. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 211
  13. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 208
  14. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 197
  15. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 195
  16. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 175
  17. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 166
  18. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
  19. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
  20. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 128
  21. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 124
  22. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 114
  23. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 111
  24. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  25. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
  26. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70
  27. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 52
  28. 2024Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Trackingcited by 48
  29. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  30. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 44
  31. 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 34
  32. 2022Egocentric Prediction of Action Target in 3Dcited by 31
  33. 2024Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videoscited by 21
  34. 2024MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videoscited by 18
  35. 2025The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimationcited by 16
  36. 2024HandsOnVLM: Vision-Language Models for Hand-Object Interaction Predictioncited by 15
  37. 2024EMAG: Ego-motion Aware and Generalizable 2D Hand Forecasting from Egocentric Videoscited by 8
  38. 2025Novel Diffusion Models for Multimodal 3D Hand Trajectory Predictioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026ObjectForesight: Predicting Future 3D Object Trajectories from Human Videoscited by 5
  2. 2026Robot Learning from Human Videos: A Surveycited by 5
  3. 2026BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Modelscited by 1
  4. 2026MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Predictioncited by 1
  5. 2026EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generationcited by 0
  6. 2026Unified Video-Action Joint Denoising for Dexterous Action and Data Generationcited by 0
  7. 2026Human-Centric Intelligence in the Era of Foundation Models: A Surveycited by 0
  8. 2026From Where to How: Continuous 4D Interaction Forecasting from Egocentric Videocited by 0