mlsanityData to 7 Oct 2026

Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting

arXiv 2307.08243 ↗First posted 17 Jul 2023cs.CVcited by 33

Not on the map. Why?

Abstract (arXiv)

Hand trajectory forecasting from egocentric views is crucial for enabling a prompt understanding of human intentions when interacting with AR/VR systems. However, existing methods handle this problem in a 2D image space which is inadequate for 3D real-world applications. In this paper, we set up an egocentric 3D hand trajectory forecasting task that aims to predict hand trajectories in a 3D space from early observed RGB videos in a first-person view. To fulfill this goal, we propose an uncertainty-aware state space Transformer (USST) that takes the merits of the attention mechanism and aleatoric uncertainty within the framework of the classical state-space model. The model can be further enhanced by the velocity constraint and visual prompt tuning (VPT) on large vision transformers. Moreover, we develop an annotation workflow to collect 3D hand trajectories with high quality. Experimental results on H2O and EgoPAT3D datasets demonstrate the superiority of USST for both 2D and 3D trajectory forecasting. The code and datasets are publicly released: https://actionlab-cv.github.io/EgoHandTrajPred.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

42 of 55 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  5. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  6. 2022Visual Prompt Tuningcited by 1,674
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  9. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  10. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  11. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  12. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 327
30 more
  1. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  2. 2018Convolutional Social Pooling for Vehicle Trajectory Predictioncited by 221
  3. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  4. 2021Anticipative Video Transformercited by 197
  5. 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
  6. 2015Deep Kalman Filterscited by 163
  7. 2022NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimationcited by 151
  8. 2018Bounding Box Regression with Uncertainty for Accurate Object Detectioncited by 143
  9. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  10. 2020Dynamical Variational Autoencoders: A Comprehensive Reviewcited by 127
  11. 2019Learning the Depths of Moving People by Watching Frozen Peoplecited by 120
  12. 2018Disentangled Sequential Autoencodercited by 117
  13. 2016Sequential Neural Models with Stochastic Layerscited by 115
  14. 2019SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Predictioncited by 111
  15. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  16. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  17. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 98
  18. 2020Uncertainty-based Traffic Accident Anticipation with Spatio-Temporal Relational Learningcited by 91
  19. 2019Ego-Pose Estimation and Forecasting as Real-Time PD Controlcited by 87
  20. 2021Multi-Person 3D Motion Prediction with Multi-Range Transformerscited by 63
  21. 2014Learning Stochastic Recurrent Networkscited by 60
  22. 2021Consistent Depth of Moving Objects in Videocited by 56
  23. 2017Using a single RGB frame for real time 3D hand pose estimation in the wildcited by 39
  24. 2019A Recurrent Variational Autoencoder for Speech Enhancementcited by 38
  25. 2022Egocentric Prediction of Action Target in 3Dcited by 31
  26. 2019Analyzing the Variety Loss in the Context of Probabilistic Trajectory Predictioncited by 30
  27. 2020Symbiotic Attention with Privileged Information for Egocentric Action Recognitioncited by 27
  28. 2021Egocentric Human Trajectory Forecasting with a Wearable Camera and Multi-Modal Fusioncited by 17
  29. 2020Forecasting Characteristic 3D Poses of Human Actionscited by 16
  30. 2019NEMO: Future Object Localization Using Noisy Ego Priorscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 14; 2026: 10.

33 papers.

Sort:

By area:

  1. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  2. 2024Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videoscited by 21
  3. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  4. 2024MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videoscited by 17
  5. 2024EAGLE: Egocentric AGgregated Language-video Enginecited by 16
  6. 2025The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimationcited by 16
  7. 2024HandsOnVLM: Vision-Language Models for Hand-Object Interaction Predictioncited by 15
  8. 2024AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 14
  9. 2025Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Visioncited by 11
  10. 2024PEAR: Phrase-Based Hand-Object Interaction Anticipationcited by 10
  11. 2025Novel Diffusion Models for Multimodal 3D Hand Trajectory Predictioncited by 7
  12. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactionscited by 7
21 more
  1. 2024Bidirectional Progressive Transformer for Interaction Intention Anticipationcited by 6
  2. 2025Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videoscited by 6
  3. 2026AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignmentcited by 5
  4. 2025Uni-Hand: Universal Hand Motion Forecasting in Egocentric Viewscited by 4
  5. 2023Generative Hierarchical Temporal Transformer for Hand Pose and Action Modelingcited by 3
  6. 2025Zero-Shot Temporal Interaction Localization for Egocentric Videoscited by 2
  7. 2025Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Spancited by 2
  8. 2025Bimanual 3D Hand Motion and Articulation Forecasting in Everyday Imagescited by 1
  9. 2025SFHand: Learning Embodied Manipulation by Streaming Egocentric 3D Hand Forecastingcited by 1
  10. 2026Integrating Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 1
  11. 2026Spatially Prompted Visual Trajectory Prediction for Egocentric Manipulationcited by 1
  12. 2025EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videoscited by 0
  13. 2025Uncertainty Aware-Predictive Control Barrier Functions: Safer Human Robot Interaction through Probabilistic Motion Forecastingcited by 0
  14. 2025Seeing My Future: Predicting Situated Interaction Behavior in Virtual Realitycited by 0
  15. 2026Int3DNet: Scene-Motion Cross Attention Network for 3D Intention Prediction in Mixed Realitycited by 0
  16. 2026EggHand: A Multimodal Foundation Model for Egocentric Hand Pose Forecastingcited by 0
  17. 2026Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecastingcited by 0
  18. 2026From Where to How: Continuous 4D Interaction Forecasting from Egocentric Videocited by 0
  19. 2026EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Datasetcited by 0
  20. 2026HAP: A Hand-Driven Active Perception Framework for Egocentric Head Motion Predictioncited by 0
  21. 2026AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Ego-centric Predictive Model Conditioned on Hand Trajectories
  2. 2026Ego-Human Motion Prediction with 3D-Aware LLM
  3. 2026EgoTactile: Learning Grasp Pressure for Everyday Objects from Egocentric Video
  4. 2026ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery
  5. 2026ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wild
  6. 2026EMPIRE: Explicit Manipulation Planning as a Learnable Intermediate Representation for Egocentric Hand-Motion Forecasting
  7. 2025MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation
  8. 2026Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gestures