# Ego-Exo 3D Hand Tracking in the Wild with a Mobile Multi-Camera Rig

arXiv: https://arxiv.org/abs/2510.02601 · HTML: https://mlsanity.com/paper/2510.02601/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 2 Oct 2025.

## Builds on (most cited first, 21 of 21)

- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [Sapiens: Foundation for Human Vision Models](https://mlsanity.com/paper/2408.12569/) (2024)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos](https://mlsanity.com/paper/2411.19167/) (2024)
- [Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild](https://mlsanity.com/paper/2406.09905/) (2024)
- [TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding](https://mlsanity.com/paper/2401.08399/) (2024)
- [OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion](https://mlsanity.com/paper/2403.19417/) (2024)
- [mm-Pose: Real-Time Human Skeletal Posture Estimation using mmWave Radars and CNNs](https://mlsanity.com/paper/1911.09592/) (2019)
- [UmeTrack: Unified multi-view end-to-end hand tracking for VR](https://mlsanity.com/paper/2211.00099/) (2022)
- [GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities](https://mlsanity.com/paper/2412.04244/) (2024)
- [3DRIMR: 3D Reconstruction and Imaging via mmWave Radar based on Deep Learning](https://mlsanity.com/paper/2108.02858/) (2021)
- [An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing](https://mlsanity.com/paper/1712.03084/) (2017)
- [Quadric Representations for LiDAR Odometry, Mapping and Localization](https://mlsanity.com/paper/2304.14190/) (2023)
- [MLPHand: Real Time Multi-View 3D Hand Mesh Reconstruction via MLP Modeling](https://mlsanity.com/paper/2406.16137/) (2024)
- [Learning to Search on Manifolds for 3D Pose Estimation of Articulated Objects](https://mlsanity.com/paper/1612.00596/) (2016)

## Built on by (most cited first, 1 of 1)

- [EgoWholeView: Camera-Only Head-Mounted Capture of Body and Bimanual Hand Motion for Embodied Demonstrations](https://mlsanity.com/paper/2610.16256/) (2026)

## Abstract (arXiv)

Accurate 3D tracking of hands and their interactions with the world in unconstrained settings remains a significant challenge for egocentric computer vision. With few exceptions, existing datasets are predominantly captured in controlled lab setups, limiting environmental diversity and model generalization. To address this, we introduce a novel marker-less multi-camera system designed to capture precise 3D hands and objects, which allows for nearly unconstrained mobility in genuinely in-the-wild conditions. We combine a lightweight, back-mounted capture rig with eight exocentric cameras, and a user-worn Meta Quest 3 headset, which contributes two egocentric views. We design an ego-exo tracking pipeline to generate accurate 3D hand pose ground truth from this system, and rigorously evaluate its quality. By collecting an annotated dataset featuring synchronized multi-view images and precise 3D hand poses, we demonstrate the capability of our approach to significantly reduce the trade-off between environmental realism and 3D annotation accuracy.