mlsanityData to 7 Oct 2026

Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos

arXiv 2204.01696 ↗First posted 4 Apr 2022cs.CV cs.LGcited by 110

Not on the map. Why?

Abstract (arXiv)

We propose to forecast future hand-object interactions given an egocentric video. Instead of predicting action labels or pixels, we directly predict the hand motion trajectory and the future contact points on the next active object (i.e., interaction hotspots). This relatively low-dimensional representation provides a concrete description of future interactions. To tackle this task, we first provide an automatic way to collect trajectory and hotspots labels on large-scale data. We then use this data to train an Object-Centric Transformer (OCT) model for prediction. Our model performs hand and object interaction reasoning via the self-attention mechanism in Transformers. OCT also provides a probabilistic framework to sample the future trajectory and hotspots to handle uncertainty in prediction. We perform experiments on the Epic-Kitchens-55, Epic-Kitchens-100, and EGTEA Gaze+ datasets, and show that OCT significantly outperforms state-of-the-art approaches by a large margin. Project page is available at https://stevenlsw.github.io/hoi-forecast .

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 61 of its 94 references that are in the map. First posted on arXiv 4 Apr 2022.

Builds on

61 of 94 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  10. 2016Layer Normalizationcited by 5,579
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2021Video Swin Transformercited by 1,090
  8. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  9. 2016Simple Online and Realtime Trackingcited by 862
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  11. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  12. 2020Rescaling Egocentric Visioncited by 529
  13. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  14. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  15. 2020Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Predictioncited by 324
  16. 2018Video Action Transformer Networkcited by 316
  17. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  18. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  19. 2020It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Predictioncited by 235
  20. 2020Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Predictioncited by 218
  21. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  22. 2021Anticipative Video Transformercited by 197
  23. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
  24. 2016What do different evaluation metrics tell us about saliency models?cited by 184
  25. 2020Transformer Networks for Trajectory Forecastingcited by 180
  26. 2020From Goals, Waypoints & Paths To Long Term Human Trajectory Forecastingcited by 166
  27. 2021Towards Long-Form Video Understandingcited by 146
  28. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  29. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  30. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 126
  31. 2020Rolling-Unrolling LSTMs for Action Anticipation from First-Person Videocited by 119
  32. 2019Peeking into the Future: Predicting Future Person Activities and Locations in Videoscited by 112
  33. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  34. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  35. 2019Wasserstein Dependency Measure for Representation Learningcited by 69
  36. 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
  37. 2017Future Person Localization in First-Person Videoscited by 57
  38. 2020Goal-GAN: Multimodal Trajectory Prediction Based on Goal Position Estimationcited by 57

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 27; 2024: 30; 2025: 28; 2026: 20.

110 papers.

Sort:

By area:

  1. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  2. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  3. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
  4. 2023Structured World Models from Human Videoscited by 144
  5. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  6. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  7. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  8. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  9. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  10. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  11. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  12. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
38 more
  1. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  2. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
  3. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 66
  4. 2023LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Groundingcited by 64
  5. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 55
  6. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  7. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  8. 2023Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignmentcited by 46
  9. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  10. 2023One-Shot Open Affordance Learning with Foundation Modelscited by 45
  11. 2024G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesiscited by 43
  12. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  13. 2022Interaction Region Visual Transformer for Egocentric Action Anticipationcited by 36
  14. 2023Affordance Grounding from Demonstration Video to Target Imagecited by 34
  15. 2024Learning Precise Affordances from Egocentric Videos for Robotic Manipulationcited by 34
  16. 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 33
  17. 2023EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understandingcited by 32
  18. 2025HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Modelscited by 32
  19. 2023Helping Hands: An Object-Aware Ego-Centric Video Recognition Modelcited by 31
  20. 2023Pretrained Language Models as Visual Planners for Human Assistancecited by 28
  21. 2023LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuningcited by 28
  22. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  23. 2025ViPRA: Video Prediction for Robot Actionscited by 24
  24. 2022Grounded Affordance from Exocentric Viewcited by 21
  25. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  26. 2024Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videoscited by 21
  27. 2024Controlling the World by Sleight of Handcited by 19
  28. 2023Multi-label affordance mapping from egocentric visioncited by 18
  29. 2024MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videoscited by 17
  30. 2024GEAL: Generalizable 3D Affordance Learning with Cross-Modal Consistencycited by 17
  31. 2023Leveraging Next-Active Objects for Context-Aware Anticipation in Egocentric Videoscited by 16
  32. 2025The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimationcited by 16
  33. 2024HandsOnVLM: Vision-Language Models for Hand-Object Interaction Predictioncited by 15
  34. 2024AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 14
  35. 2025GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulationcited by 14
  36. 2024STEER: Flexible Robotic Manipulation via Dense Language Groundingcited by 13
  37. 2025MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videoscited by 12
  38. 2025Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Visioncited by 11

50 most cited of 110 All 110 (JSON)