# EventEgo3D: 3D Human Motion Capture from Egocentric Event Streams

arXiv: https://arxiv.org/abs/2404.08640 · HTML: https://mlsanity.com/paper/2404.08640/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 52 references that are in the map.
- First posted on arXiv 12 Apr 2024.

## Builds on (most cited first, 30 of 37)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Learning to Estimate 3D Human Pose and Shape from a Single Color Image](https://mlsanity.com/paper/1805.04092/) (2018)
- [Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception](https://mlsanity.com/paper/2306.06362/) (2023)
- [Video to Events: Recycling Video Datasets for Event Cameras](https://mlsanity.com/paper/1912.03095/) (2019)
- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [BlazePose: On-device Real-time Body Pose tracking](https://mlsanity.com/paper/2006.10204/) (2020)
- [Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation](https://mlsanity.com/paper/2106.05969/) (2021)
- [Ego-Body Pose Estimation via Ego-Head Pose Estimation](https://mlsanity.com/paper/2212.04636/) (2022)
- [EventNeRF: Neural Radiance Fields from a Single Colour Event Camera](https://mlsanity.com/paper/2206.11896/) (2022)
- [Ego-Pose Estimation and Forecasting as Real-Time PD Control](https://mlsanity.com/paper/1906.03173/) (2019)
- [xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera](https://mlsanity.com/paper/1907.10045/) (2019)
- [Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera](https://mlsanity.com/paper/1803.05959/) (2018)
- [EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera](https://mlsanity.com/paper/1908.11505/) (2019)
- [EgoHumans: An Egocentric 3D Multi-Human Benchmark](https://mlsanity.com/paper/2305.16487/) (2023)
- [UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture](https://mlsanity.com/paper/2208.01633/) (2022)
- [Estimating Egocentric 3D Human Pose in Global Space](https://mlsanity.com/paper/2104.13454/) (2021)
- [SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera](https://mlsanity.com/paper/2011.01519/) (2020)
- [EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream](https://mlsanity.com/paper/2012.06475/) (2020)
- [EventHPE: Event-based 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2108.06819/) (2021)
- [Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision](https://mlsanity.com/paper/2201.07929/) (2022)
- [Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Views](https://mlsanity.com/paper/2304.06024/) (2023)
- [Scene-aware Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2212.11684/) (2022)
- [Efficient Human Pose Estimation via 3D Event Point Cloud](https://mlsanity.com/paper/2206.04511/) (2022)
- [EgoCap: Egocentric Marker-less Motion Capture with Two Fisheye Cameras (Extended Abstract)](https://mlsanity.com/paper/1701.00142/) (2016)

## Built on by (most cited first, 29 of 29)

- [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024)
- [Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding](https://mlsanity.com/paper/2503.09143/) (2025)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs](https://mlsanity.com/paper/2408.17168/) (2024)
- [E-3DGS: Event-Based Novel View Rendering of Large-Scale Scenes Using 3D Gaussian Splatting](https://mlsanity.com/paper/2502.10827/) (2025)
- [Helios: An extremely low power event-based gesture recognition for always-on smart eyewear](https://mlsanity.com/paper/2407.05206/) (2024)
- [Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions](https://mlsanity.com/paper/2508.04681/) (2025)
- [Bring Your Rear Cameras for Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2503.11652/) (2025)
- [EventEgo3D++: 3D Human Motion Capture from a Head-Mounted Event Camera](https://mlsanity.com/paper/2502.07869/) (2025)
- [EventEgoHands: Event-based Egocentric 3D Hand Mesh Reconstruction](https://mlsanity.com/paper/2505.19169/) (2025)
- [Human Motion Capture from Loose and Sparse Inertial Sensors with Garment-aware Diffusion Models](https://mlsanity.com/paper/2506.15290/) (2025)
- [Dynamic EventNeRF: Reconstructing General Dynamic Scenes from Multi-view RGB and Event Streams](https://mlsanity.com/paper/2412.06770/) (2024)
- [EgoEvGesture: Gesture Recognition Based on Egocentric Event Camera](https://mlsanity.com/paper/2503.12419/) (2025)
- [From Sharp to Blur: Unsupervised Domain Adaptation for 2D Human Pose Estimation Under Extreme Motion Blur Using Event Cameras](https://mlsanity.com/paper/2507.22438/) (2025)
- [EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera](https://mlsanity.com/paper/2605.12498/) (2026)
- [Reading in the Dark with Foveated Event Vision](https://mlsanity.com/paper/2506.06918/) (2025)
- [EgoMusic-driven Human Dance Motion Estimation with Skeleton Mamba](https://mlsanity.com/paper/2508.10522/) (2025)
- [EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras](https://mlsanity.com/paper/2605.12297/) (2026)
- [Helios 2.0: A Robust, Ultra-Low Power Gesture Recognition System Optimised for Event-Sensor based Wearables](https://mlsanity.com/paper/2503.07825/) (2025)
- [Event-based Egocentric Human Pose Estimation in Dynamic Environment](https://mlsanity.com/paper/2505.22007/) (2025)
- [Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reach](https://mlsanity.com/paper/2512.16456/) (2025)
- [E-3DPSM: A State Machine for Event-Based Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2604.08543/) (2026)
- [Mocap Anywhere: Towards Pairwise-Distance based Motion Capture in the Wild (for the Wild)](https://mlsanity.com/paper/2601.19519/) (2026)
- [EventNeuS: 3D Mesh Reconstruction from a Single Event Camera](https://mlsanity.com/paper/2602.03847/) (2026)
- [EgoPoseVR: Spatiotemporal Multi-Modal Reasoning for Egocentric Full-Body Pose in Virtual Reality](https://mlsanity.com/paper/2602.05590/) (2026)
- [Understanding From Human Perspective: A Multi-agent System for Interactive Egocentric Medical Image Segmentation](https://mlsanity.com/paper/2607.17341/) (2026)
- [Cooking beyond Frames: A Stereo Event Camera Dataset in the Kitchen](https://mlsanity.com/paper/2608.04865/) (2026)
- [Seeing the World and the Self from Egocentric Video](https://mlsanity.com/paper/2609.01276/) (2026)
- [EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Dataset](https://mlsanity.com/paper/2609.17189/) (2026)

## Abstract (arXiv)

Monocular egocentric 3D human motion capture is a challenging and actively researched problem. Existing methods use synchronously operating visual sensors (e.g. RGB cameras) and often fail under low lighting and fast motions, which can be restricting in many applications involving head-mounted devices. In response to the existing limitations, this paper 1) introduces a new problem, i.e., 3D human motion capture from an egocentric monocular event camera with a fisheye lens, and 2) proposes the first approach to it called EventEgo3D (EE3D). Event streams have high temporal resolution and provide reliable cues for 3D human motion capture under high-speed human motions and rapidly changing illumination. The proposed EE3D framework is specifically tailored for learning with event streams in the LNES representation, enabling high 3D reconstruction accuracy. We also design a prototype of a mobile head-mounted device with an event camera and record a real dataset with event observations and the ground-truth 3D human poses (in addition to the synthetic dataset). Our EE3D demonstrates robustness and superior 3D accuracy compared to existing solutions across various challenging experiments while supporting real-time 3D pose update rates of 140Hz.