mlsanityData to 7 Oct 2026

HybridCap: Inertia-aid Monocular Capture of Challenging Human Motions

arXiv 2203.09287 ↗First posted 17 Mar 2022cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Monocular 3D motion capture (mocap) is beneficial to many applications. The use of a single camera, however, often fails to handle occlusions of different body parts and hence it is limited to capture relatively simple movements. We present a light-weight, hybrid mocap technique called HybridCap that augments the camera with only 4 Inertial Measurement Units (IMUs) in a learning-and-optimization framework. We first employ a weakly-supervised and hierarchical motion inference module based on cooperative Gated Recurrent Unit (GRU) blocks that serve as limb, body and root trackers as well as an inverse kinematics solver. Our network effectively narrows the search space of plausible motions via coarse-to-fine pose estimation and manages to tackle challenging movements with high efficiency. We further develop a hybrid optimization scheme that combines inertial feedback and visual cues to improve tracking accuracy. Extensive experiments on various datasets demonstrate HybridCap can robustly handle challenging movements ranging from fitness actions to Latin dance. It also achieves real-time performance up to 60 fps with state-of-the-art accuracy.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 37 of its 79 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

37 of 79 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  3. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  4. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  5. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  6. 2017End-to-end Recovery of Human Shape and Posecited by 883
  7. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  8. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  9. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  10. 2016Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervisioncited by 493
  11. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  12. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
25 more
  1. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  2. 2018Learning 3D Human Dynamics from Videocited by 315
  3. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  4. 2018Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodiescited by 239
  5. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  6. 2017Unite the People: Closing the Loop Between 3D and 2D Human Representationscited by 205
  7. 2020STAR: Sparse Trained Articulated Human Body Regressorcited by 194
  8. 2019DeepHuman: 3D Human Reconstruction from a Single Imagecited by 182
  9. 2018Monocular Total Capture: Posing Face, Body, and Hands in the Wildcited by 162
  10. 2020DeepCap: Monocular Human Performance Capture Using Weak Supervisioncited by 162
  11. 2021Probabilistic Modeling for Human Mesh Recoverycited by 152
  12. 2022Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensorscited by 144
  13. 2017Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUscited by 108
  14. 2018LiveCap: Real-time Human Performance Capture from Monocular Videocited by 93
  15. 2017Towards Accurate Markerless Human Shape and Pose Estimation over Timecited by 92
  16. 2021Neural Monocular 3D Human Motion Capture with Physical Awarenesscited by 70
  17. 2019EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Cameracited by 65
  18. 2017MonoPerfCap: Human Performance Capture from Monocular Videocited by 63
  19. 2020Neural Descent for Visual 3D Human Pose and Shapecited by 60
  20. 2017Harvesting Multiple Views for Marker-less 3D Human Pose Annotationscited by 48
  21. 2020Fusing Wearable IMUs with Multi-View Images for Human Pose Estimation: A Geometric Approachcited by 27
  22. 2021ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal Referencescited by 26
  23. 2016FlyCap: Markerless Motion Capture Using Multiple Autonomous Flying Camerascited by 23
  24. 2016Model-based Outdoor Performance Capturecited by 13
  25. 2020Revisiting the Continuity of Rotation Representations in Neural Networkscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 5; 2025: 5; 2026: 2.

19 papers.

Sort:

By area:

  1. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactionscited by 205
  2. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  3. 2023OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllerscited by 47
  4. 2022LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensorscited by 43
  5. 2023I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactionscited by 42
  6. 2022Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDARcited by 23
  7. 2023Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capturecited by 17
  8. 2023IKOL: Inverse kinematics optimization layer for 3D human pose and shape estimation via Gauss-Newton differentiationcited by 9
  9. 2025Motion Generation: A Survey of Generative Approaches and Benchmarkscited by 8
  10. 2024StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offsetcited by 5
  11. 2025UMotion: Uncertainty-driven Human Motion Estimation from Inertial and Ultra-wideband Unitscited by 5
  12. 2024Motion Capture from Inertial and Vision Sensorscited by 4
7 more
  1. 2025Mojito: LLM-Aided Motion Instructor with Jitter-Reduced Inertial Tokenscited by 4
  2. 2024FreeCap: Hybrid Calibration-Free Motion Capture in Open Environmentscited by 3
  3. 2025DiffCap: Diffusion-based Real-time Human Motion Capture using Sparse IMUs and a Monocular Cameracited by 2
  4. 2026Stereo-Inertial Poser: Towards Metric-Accurate Shape-Aware Motion Capture Using Sparse IMUs and a Single Stereo Cameracited by 1
  5. 2024Learning Human Motion from Monocular Videos via Cross-Modal Manifold Alignmentcited by 0
  6. 2025Shape-aware Inertial Poser: Motion Tracking for Humans with Diverse Shapes Using Sparse Inertial Sensorscited by 0
  7. 2026IMU-HOI: A Symbiotic Framework for Coherent Human-Object Interaction and Motion Capture via Contact-Conscious Inertial Fusioncited by 0