mlsanityData to 14 Oct 2026

3D Pose Estimation of Two Interacting Hands from a Monocular Event Camera

arXiv 2312.14157 ↗First posted 21 Dec 2023cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

3D hand tracking from a monocular video is a very challenging problem due to hand interactions, occlusions, left-right hand ambiguity, and fast motion. Most existing methods rely on RGB inputs, which have severe limitations under low-light conditions and suffer from motion blur. In contrast, event cameras capture local brightness changes instead of full image frames and do not suffer from the described effects. Unfortunately, existing image-based techniques cannot be directly applied to events due to significant differences in the data modalities. In response to these challenges, this paper introduces the first framework for 3D tracking of two fast-moving and interacting hands from a single monocular event camera. Our approach tackles the left-right hand ambiguity with a novel semi-supervised feature-wise attention mechanism and integrates an intersection loss to fix hand collisions. To facilitate advances in this research domain, we release a new synthetic large-scale dataset of two interacting hands, Ev2Hands-S, and a new real benchmark with real event streams and ground-truth 3D annotations, Ev2Hands-R. Our approach outperforms existing methods in terms of the 3D reconstruction accuracy and generalises to real data under severe light conditions.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 49 of its 68 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

49 of 68 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2019High Speed and High Dynamic Range Video with an Event Cameracited by 345
  7. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
  8. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  9. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  10. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  11. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
  12. 2018Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Carscited by 200
37 more
  1. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  2. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  3. 2019Video to Events: Recycling Video Datasets for Event Camerascited by 165
  4. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  5. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  6. 2020Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancycited by 125
  7. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  8. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 123
  9. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  10. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  11. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  12. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  13. 2022EventNeRF: Neural Radiance Fields from a Single Colour Event Cameracited by 92
  14. 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
  15. 2017Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensorcited by 84
  16. 2022ESS: Learning Event-based Semantic Segmentation from Still Imagescited by 82
  17. 2021Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Cameracited by 70
  18. 2018Continuous-time Intensity Estimation Using Event Camerascited by 68
  19. 2019EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Cameracited by 65
  20. 2022LISA: Learning Implicit Shape and Appearance of Handscited by 63
  21. 2020NeuralAnnot: Neural Annotator for 3D Human Mesh Training Setscited by 62
  22. 2021E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognitioncited by 50
  23. 2022Event Transformer. A sparse-aware solution for efficient event data processingcited by 50
  24. 2020DeepHandMesh: A Weakly-supervised Deep Encoder-Decoder Framework for High-fidelity Hand Mesh Modelingcited by 49
  25. 2020EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Streamcited by 48
  26. 2017BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysiscited by 46
  27. 2021Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentationcited by 40
  28. 2023Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wildcited by 36
  29. 2016Fast and Robust Hand Tracking Using Detection-Guided Optimizationcited by 30
  30. 2022State of the Art in Dense Monocular Non-Rigid 3D Reconstructioncited by 30
  31. 2022Efficient Human Pose Estimation via 3D Event Point Cloudcited by 25
  32. 2020HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation from a Single Depth Mapcited by 23
  33. 2021Differentiable Event Stream Simulator for Non-Rigid 3D Trackingcited by 23
  34. 2018Point-to-Pose Voting based Hand Pose Estimation using Residual Permutation Equivariant Layercited by 21
  35. 2023Decaf: Monocular Deformation Capture for Face and Hand Interactionscited by 18
  36. 2022Event-based Non-Rigid Reconstruction from Contourscited by 17
  37. 2022HandFlow: Quantifying View-Dependent 3D Ambiguity in Two-Hand Reconstruction with Normalizing Flowcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 5.

17 papers.

Sort:

  1. 2024EventEgo3D: 3D Human Motion Capture from Egocentric Event Streamscited by 29
  2. 2024Recent Trends in 3D Reconstruction of General Non-Rigid Scenescited by 28
  3. 2025E-3DGS: Event-Based Novel View Rendering of Large-Scale Scenes Using 3D Gaussian Splattingcited by 9
  4. 2025EventEgo3D++: 3D Human Motion Capture from a Head-Mounted Event Cameracited by 6
  5. 2025A Survey of 3D Reconstruction with Event Camerascited by 5
  6. 2025EventEgoHands: Event-based Egocentric 3D Hand Mesh Reconstructioncited by 5
  7. 2024Dynamic EventNeRF: Reconstructing General Dynamic Scenes from Multi-view RGB and Event Streamscited by 4
  8. 2025EgoEvGesture: Gesture Recognition Based on Egocentric Event Cameracited by 4
  9. 2024Event-based Motion Deblurring via Multi-Temporal Granularity Fusioncited by 2
  10. 2025A Survey on Event-driven 3D Reconstruction: Development under Different Categoriescited by 2
  11. 2026EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Camerascited by 2
  12. 2025EvHand-FPV: Efficient Event-Based 3D Hand Tracking from First-Person Viewcited by 1
5 more
  1. 20256D Pose Estimation on Spoons and Handscited by 0
  2. 2026EventNeuS: 3D Mesh Reconstruction from a Single Event Cameracited by 0
  3. 2026Event-based Visual Deformation Measurementcited by 0
  4. 2026Cooking beyond Frames: A Stereo Event Camera Dataset in the Kitchencited by 0
  5. 2026EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Datasetcited by 0