# Hybrid 3D Human Pose Estimation with Monocular Video and Sparse IMUs

arXiv: https://arxiv.org/abs/2404.17837 · HTML: https://mlsanity.com/paper/2404.17837/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 55 references that are in the map.
- First posted on arXiv 27 Apr 2024.

## Builds on (most cited first, 3 of 3)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [P-STMO: Pre-Trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimation](https://mlsanity.com/paper/2203.07628/) (2022)
- [FusePose: IMU-Vision Sensor Fusion in Kinematic Space for Parametric Human Pose Estimation](https://mlsanity.com/paper/2208.11960/) (2022)

## Built on by (most cited first, 2 of 2)

- [VIMCAN: Visual-Inertial 3D Human Pose Estimation with Hybrid Mamba-Cross-Attention Network](https://mlsanity.com/paper/2605.07552/) (2026)
- [Paving the Way Towards Kinematic Assessment Using Monocular Video: A Preclinical Benchmark of State-of-the-Art Deep-Learning-Based 3D Human Pose Estimators Against Inertial Sensors in Daily Living Activities](https://mlsanity.com/paper/2510.02264/) (2025)

## Abstract (arXiv)

Temporal 3D human pose estimation from monocular videos is a challenging task in human-centered computer vision due to the depth ambiguity of 2D-to-3D lifting. To improve accuracy and address occlusion issues, inertial sensor has been introduced to provide complementary source of information. However, it remains challenging to integrate heterogeneous sensor data for producing physically rational 3D human poses. In this paper, we propose a novel framework, Real-time Optimization and Fusion (RTOF), to address this issue. We first incorporate sparse inertial orientations into a parametric human skeleton to refine 3D poses in kinematics. The poses are then optimized by energy functions built on both visual and inertial observations to reduce the temporal jitters. Our framework outputs smooth and biomechanically plausible human motion. Comprehensive experiments with ablation studies demonstrate its rationality and efficiency. On Total Capture dataset, the pose estimation error is significantly decreased compared to the baseline method.