mlsanityData to 14 Oct 2026

UmeTrack: Unified multi-view end-to-end hand tracking for VR

arXiv 2211.00099 ↗First posted 31 Oct 2022cs.CVcited by 50

Not on the map. Why?

Abstract (arXiv)

Real-time tracking of 3D hand pose in world space is a challenging problem and plays an important role in VR interaction. Existing work in this space are limited to either producing root-relative (versus world space) 3D pose or rely on multiple stages such as generating heatmaps and kinematic optimization to obtain 3D pose. Moreover, the typical VR scenario, which involves multi-view tracking from wide acfov cameras is seldom addressed by these methods. In this paper, we present a unified end-to-end differentiable framework for multi-view, multi-frame hand tracking that directly predicts 3D hand pose in world space. We demonstrate the benefits of end-to-end differentiabilty by extending our framework with downstream tasks such as jitter reduction and pinch prediction. To demonstrate the efficacy of our model, we further present a new large-scale egocentric hand pose dataset that consists of both real and synthetic data. Experiments show that our system trained on this dataset handles various challenging interactive motions, and has been successfully applied to real-time VR applications.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

23 of 38 references.

References · 23

  1. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  2. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  3. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  4. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  5. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  6. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
  7. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  8. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 181
  9. 2019Learnable Triangulation of Human Posecited by 161
  10. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  11. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  12. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
11 more
  1. 2019Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Imagecited by 132
  2. 2020Epipolar Transformerscited by 117
  3. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  4. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  5. 2018Cross-modal Deep Variational Hand Pose Estimationcited by 89
  6. 2020Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representationcited by 57
  7. 2016Deep Kinematic Pose Regressioncited by 40
  8. 2017Interpretable Transformations with Encoder-Decoder Networkscited by 34
  9. 2020SeqHAND:RGB-Sequence-Based 3D Hand Pose and Shape Estimationcited by 34
  10. 2020PCLs: Geometry-aware Neural Reconstruction of 3D Pose with Perspective Crop Layerscited by 16
  11. 2020MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 14; 2025: 7; 2026: 16.

50 papers.

Sort:

By area:

  1. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 208
  2. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 129
  3. 2023AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimationcited by 64
  4. 2024Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Trackingcited by 48
  5. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  6. 2023EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywherecited by 34
  7. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 33
  8. 2024emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimationcited by 22
  9. 2024Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videoscited by 21
  10. 2024Single-to-Dual-View Adaptation for Egocentric 3D Hand Pose Estimationcited by 19
  11. 2024EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Visioncited by 19
  12. 2025BOP Challenge 2024 on Model-Based and Model-Free 6D Object Pose Estimationcited by 19
38 more
  1. 2024On the Utility of 3D Hand Poses for Action Recognitioncited by 17
  2. 2023Novel-view Synthesis and Pose Estimation for Hand-Object Interaction from Sparse Viewscited by 16
  3. 2023A2J-Transformer: Anchor-to-Joint Transformer Network for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 15
  4. 2023PressureVision++: Estimating Fingertip Pressure from Diverse RGB Imagescited by 12
  5. 2025From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Modelscited by 11
  6. 2023POEM: Reconstructing Hand in a Point Embedded Multi-view Stereocited by 10
  7. 2023Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Imagescited by 10
  8. 2023HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learningcited by 7
  9. 2025SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-trainingcited by 7
  10. 2024Multi-view Hand Reconstruction with a Point-Embedded Transformercited by 6
  11. 2025Uni-Hand: Universal Hand Motion Forecasting in Egocentric Viewscited by 5
  12. 2024TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Visioncited by 4
  13. 2026OnlineHMR: Video-based Online World-Grounded Human Mesh Recoverycited by 4
  14. 2026SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wildcited by 4
  15. 2023OO-dMVMT: A Deep Multi-view Multi-task Classification Framework for Real-time 3D Hand Gesture Classification and Segmentationcited by 3
  16. 20231st Place Solution of Egocentric 3D Hand Pose Estimation Challenge 2023 Technical Report:A Concise Pipeline for Egocentric Hand Pose Reconstructioncited by 3
  17. 2023Generative Hierarchical Temporal Transformer for Hand Pose and Action Modelingcited by 3
  18. 2026Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Glovescited by 3
  19. 2026EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimationcited by 3
  20. 2026EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Cameracited by 3
  21. 2024Pre-Training for 3D Hand Pose Estimation with Contrastive Learning on Large-Scale Hand Images in the Wildcited by 2
  22. 2026AVI-HT: Adaptive Vision-IMU Fusion for 3D Hand Trackingcited by 2
  23. 2026HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Controlcited by 2
  24. 2023UltraGlove: Hand Pose Estimation with Mems-Ultrasonic Sensorscited by 1
  25. 20241st Place Solution of Multiview Egocentric Hand Tracking Challenge ECCV2024cited by 1
  26. 2025Ego-Exo 3D Hand Tracking in the Wild with a Mobile Multi-Camera Rigcited by 1
  27. 2026EgoKit: Towards Unified Low-Cost Egocentric Data Collection with Heterogeneous Devicescited by 1
  28. 2024Continual Gesture Learning without Data via Synthetic Feature Samplingcited by 0
  29. 2025DF-Mamba: Deformable State Space Modeling for 3D Hand Pose Estimation in Interactionscited by 0
  30. 2025OMG-Bench: A New Challenging Benchmark for Skeleton-based Online Micro Hand Gesture Recognitioncited by 0
  31. 2026AirGlove: Exploring Egocentric 3D Hand Tracking and Appearance Generalization for Sensing Glovescited by 0
  32. 2026SesaHand: Enhancing 3D Hand Reconstruction via Controllable Generation with Semantic and Structural Alignmentcited by 0
  33. 2026REACH: Hand Pose Estimation from Room Cornerscited by 0
  34. 2026Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Trainingcited by 0
  35. 2026HandMvNet: Real-Time 3D Hand Pose Estimation Using Multi-View Cross-Attention Fusioncited by 0
  36. 2026ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wildcited by 0
  37. 2026EVFormer: An Egocentric Vision-EMG Bidirectional Attention Model for Bimanual Hand Pose Estimationcited by 0
  38. 2026EgoWholeView: Camera-Only Head-Mounted Capture of Body and Bimanual Hand Motion for Embodied Demonstrationscited by 0