mlsanityData to 7 Oct 2026

FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration

arXiv 2108.06428 ↗First posted 13 Aug 2021cs.CVcited by 168

Not on the map. Why?

Abstract (arXiv)

Most existing monocular 3D pose estimation approaches only focus on a single body part, neglecting the fact that the essential nuance of human motion is conveyed through a concert of subtle movements of face, hands, and body. In this paper, we present FrankMocap, a fast and accurate whole-body 3D pose estimation system that can produce 3D face, hands, and body simultaneously from in-the-wild monocular images. The core idea of FrankMocap is its modular design: We first run 3D pose regression methods for face, hands, and body independently, followed by composing the regression outputs via an integration module. The separate regression modules allow us to take full advantage of their state-of-the-art performances without compromising the original accuracy and reliability in practice. We develop three different integration modules that trade off between latency and accuracy. All of them are capable of providing simple yet effective solutions to unify the separate outputs into seamless whole-body pose estimation results. We quantitatively and qualitatively demonstrate that our modularized system outperforms both the optimization-based and end-to-end methods of estimating whole-body pose.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). Builds on 58 of its 81 references that are in the map. First posted on arXiv 13 Aug 2021.

Builds on

58 of 81 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  6. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  7. 2017End-to-end Recovery of Human Shape and Posecited by 883
  8. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  9. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  10. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  11. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  12. 2018Learning 3D Human Dynamics from Videocited by 315
38 more
  1. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 304
  2. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  3. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  4. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  5. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  6. 2018Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodiescited by 239
  7. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  8. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  9. 2018Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Networkcited by 195
  10. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  11. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  12. 2019Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervisioncited by 176
  13. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  14. 2016Panoptic Studio: A Massively Multiview System for Social Interaction Capturecited by 173
  15. 2017MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstructioncited by 163
  16. 2018Monocular Total Capture: Posing Face, Body, and Hands in the Wildcited by 162
  17. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  18. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  19. 2017VNect: Real-time 3D Human Pose Estimation with a Single RGB Cameracited by 129
  20. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  21. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  22. 2020Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimationcited by 113
  23. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  24. 2016Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Networkcited by 109
  25. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  26. 2019DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Comparecited by 100
  27. 2017Self-supervised Learning of Motion Capturecited by 97
  28. 2020Motion Guided 3D Pose Estimation from Videoscited by 85
  29. 2020Human Body Model Fitting by Learned Gradient Descentcited by 81
  30. 2021Driving-Signal Aware Full-Body Avatarscited by 81
  31. 2020Hierarchical Kinematic Human Mesh Recoverycited by 71
  32. 2020Multi-channel Transformers for Multi-articulatory Sign Language Translationcited by 71
  33. 2021Scene-aware Generative Network for Human Motion Synthesiscited by 70
  34. 20203D Human Mesh Regression with Dense Correspondencecited by 66
  35. 2021Model-based 3D Hand Reconstruction via Self-Supervised Learningcited by 65
  36. 2021Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registrationcited by 59
  37. 2020Monocular Real-time Full Body Capture with Inter-part Correlationscited by 53
  38. 2020DMD: A Large-Scale Multi-Modal Driver Monitoring Dataset for Attention and Alertness Analysiscited by 47

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 29; 2023: 45; 2024: 41; 2025: 25; 2026: 20.

168 papers.

Sort:

By area:

  1. 2023Reconstructing Hands in 3D with Transformerscited by 298
  2. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  3. 2023BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motioncited by 229
  4. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  5. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  6. 2022Human-to-Robot Imitation in the Wildcited by 157
  7. 2023SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimationcited by 154
  8. 2022PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Imagescited by 152
  9. 2021Putting People in their Place: Monocular Regression of 3D People in Depthcited by 141
  10. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 136
  11. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  12. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 128
38 more
  1. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109
  2. 2022Recovering 3D Human Mesh from Monocular Images: A Surveycited by 103
  3. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  4. 20233D Human Pose Estimation via Intuitive Physicscited by 100
  5. 2024Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shotcited by 98
  6. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  7. 2024Expressive Whole-Body 3D Gaussian Avatarcited by 89
  8. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 85
  9. 2022Capturing and Animation of Body and Clothing from Monocular Videocited by 83
  10. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 79
  11. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  12. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  13. 2022Full-Body Articulated Human-Object Interactioncited by 76
  14. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  15. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 74
  16. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  17. 2022NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactionscited by 67
  18. 2026SAM 3D Body: Robust Full-Body Human Mesh Recoverycited by 61
  19. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  20. 2023SynBody: Synthetic Dataset with Layered Human Models for 3D Human Perception and Modelingcited by 54
  21. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 52
  22. 2023EPIC Fields: Marrying 3D Geometry and Video Understandingcited by 49
  23. 2023HybrIK-X: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recoverycited by 48
  24. 2023Neural Sign Actors: A diffusion model for 3D sign language production from textcited by 48
  25. 2022Retrospectives on the Embodied AI Workshopcited by 46
  26. 2024Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mambacited by 45
  27. 2023Learning Disentangled Avatars with Hybrid 3D Representationscited by 44
  28. 2023Detecting Human-Object Contact in Imagescited by 41
  29. 2024AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimationcited by 41
  30. 2023GenZI: Zero-Shot 3D Human-Scene Interaction Generationcited by 40
  31. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  32. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 37
  33. 2024HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Datacited by 37
  34. 2023Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wildcited by 36
  35. 2023Dexterous Functional Graspingcited by 36
  36. 2025DexWild: Dexterous Human Interactions for In-the-Wild Robot Policiescited by 34
  37. 2024Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videoscited by 32
  38. 2023Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinementcited by 31

50 most cited of 168 All 168 (JSON)