# 3D Pose Estimation of Two Interacting Hands from a Monocular Event Camera

arXiv: https://arxiv.org/abs/2312.14157 · HTML: https://mlsanity.com/paper/2312.14157/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 68 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars](https://mlsanity.com/paper/1804.01310/) (2018)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [Video to Events: Recycling Video Datasets for Event Cameras](https://mlsanity.com/paper/1912.03095/) (2019)
- [GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB](https://mlsanity.com/paper/1712.01057/) (2017)
- [Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild](https://mlsanity.com/paper/2004.01946/) (2020)
- [Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy](https://mlsanity.com/paper/2009.08283/) (2020)
- [End-to-end Hand Mesh Recovery from a Monocular RGB Image](https://mlsanity.com/paper/1902.09305/) (2019)
- [Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation](https://mlsanity.com/paper/2104.14639/) (2021)
- [Capturing Hands in Action using Discriminative Salient Points and Physics Simulation](https://mlsanity.com/paper/1506.02178/) (2015)
- [Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering](https://mlsanity.com/paper/1904.04196/) (2019)
- [Hand Pose Estimation via Latent 2.5D Heatmap Regression](https://mlsanity.com/paper/1804.09534/) (2018)
- [Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data](https://mlsanity.com/paper/2003.09572/) (2020)
- [EventNeRF: Neural Radiance Fields from a Single Colour Event Camera](https://mlsanity.com/paper/2206.11896/) (2022)
- [Interacting Attention Graph for Single Image Two-Hand Reconstruction](https://mlsanity.com/paper/2203.09364/) (2022)
- [Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor](https://mlsanity.com/paper/1704.02201/) (2017)
- [ESS: Learning Event-based Semantic Segmentation from Still Images](https://mlsanity.com/paper/2203.10016/) (2022)
- [Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera](https://mlsanity.com/paper/2106.08059/) (2021)
- [Continuous-time Intensity Estimation Using Event Cameras](https://mlsanity.com/paper/1811.00386/) (2018)

## Built on by (most cited first, 17 of 17)

- [EventEgo3D: 3D Human Motion Capture from Egocentric Event Streams](https://mlsanity.com/paper/2404.08640/) (2024)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [E-3DGS: Event-Based Novel View Rendering of Large-Scale Scenes Using 3D Gaussian Splatting](https://mlsanity.com/paper/2502.10827/) (2025)
- [EventEgo3D++: 3D Human Motion Capture from a Head-Mounted Event Camera](https://mlsanity.com/paper/2502.07869/) (2025)
- [A Survey of 3D Reconstruction with Event Cameras](https://mlsanity.com/paper/2505.08438/) (2025)
- [EventEgoHands: Event-based Egocentric 3D Hand Mesh Reconstruction](https://mlsanity.com/paper/2505.19169/) (2025)
- [Dynamic EventNeRF: Reconstructing General Dynamic Scenes from Multi-view RGB and Event Streams](https://mlsanity.com/paper/2412.06770/) (2024)
- [EgoEvGesture: Gesture Recognition Based on Egocentric Event Camera](https://mlsanity.com/paper/2503.12419/) (2025)
- [Event-based Motion Deblurring via Multi-Temporal Granularity Fusion](https://mlsanity.com/paper/2412.11866/) (2024)
- [A Survey on Event-driven 3D Reconstruction: Development under Different Categories](https://mlsanity.com/paper/2503.19753/) (2025)
- [EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras](https://mlsanity.com/paper/2605.12297/) (2026)
- [EvHand-FPV: Efficient Event-Based 3D Hand Tracking from First-Person View](https://mlsanity.com/paper/2509.13883/) (2025)
- [6D Pose Estimation on Spoons and Hands](https://mlsanity.com/paper/2505.02335/) (2025)
- [EventNeuS: 3D Mesh Reconstruction from a Single Event Camera](https://mlsanity.com/paper/2602.03847/) (2026)
- [Event-based Visual Deformation Measurement](https://mlsanity.com/paper/2602.14376/) (2026)
- [Cooking beyond Frames: A Stereo Event Camera Dataset in the Kitchen](https://mlsanity.com/paper/2608.04865/) (2026)
- [EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Dataset](https://mlsanity.com/paper/2609.17189/) (2026)

## Abstract (arXiv)

3D hand tracking from a monocular video is a very challenging problem due to hand interactions, occlusions, left-right hand ambiguity, and fast motion. Most existing methods rely on RGB inputs, which have severe limitations under low-light conditions and suffer from motion blur. In contrast, event cameras capture local brightness changes instead of full image frames and do not suffer from the described effects. Unfortunately, existing image-based techniques cannot be directly applied to events due to significant differences in the data modalities. In response to these challenges, this paper introduces the first framework for 3D tracking of two fast-moving and interacting hands from a single monocular event camera. Our approach tackles the left-right hand ambiguity with a novel semi-supervised feature-wise attention mechanism and integrates an intersection loss to fix hand collisions. To facilitate advances in this research domain, we release a new synthetic large-scale dataset of two interacting hands, Ev2Hands-S, and a new real benchmark with real event streams and ground-truth 3D annotations, Ev2Hands-R. Our approach outperforms existing methods in terms of the 3D reconstruction accuracy and generalises to real data under severe light conditions.