# Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras

arXiv: https://arxiv.org/abs/2108.10378 · HTML: https://mlsanity.com/paper/2108.10378/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 66 references that are in the map.
- First posted on arXiv 23 Aug 2021.

## Builds on (most cited first, 30 of 47)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Soft-NMS -- Improving Object Detection With One Line of Code](https://mlsanity.com/paper/1704.04503/) (2017)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Convolutional Pose Machines](https://mlsanity.com/paper/1602.00134/) (2016)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop](https://mlsanity.com/paper/2103.16507/) (2021)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark](https://mlsanity.com/paper/1812.00324/) (2018)
- [Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB](https://mlsanity.com/paper/1712.03453/) (2017)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [Panoptic Studio: A Massively Multiview System for Social Interaction Capture](https://mlsanity.com/paper/1612.03153/) (2016)
- [Monocular Total Capture: Posing Face, Body, and Hands in the Wild](https://mlsanity.com/paper/1812.01598/) (2018)
- [Monocular Expressive Body Regression through Body-Driven Attention](https://mlsanity.com/paper/2008.09062/) (2020)
- [GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB](https://mlsanity.com/paper/1712.01057/) (2017)
- [Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild](https://mlsanity.com/paper/2004.01946/) (2020)
- [End-to-end Hand Mesh Recovery from a Monocular RGB Image](https://mlsanity.com/paper/1902.09305/) (2019)
- [DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation](https://mlsanity.com/paper/1511.06645/) (2015)
- [Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering](https://mlsanity.com/paper/1904.04196/) (2019)

## Built on by (most cited first, 30 of 33)

- [GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras](https://mlsanity.com/paper/2112.01524/) (2021)
- [PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images](https://mlsanity.com/paper/2207.06400/) (2022)
- [Putting People in their Place: Monocular Regression of 3D People in Depth](https://mlsanity.com/paper/2112.08274/) (2021)
- [HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling](https://mlsanity.com/paper/2204.13686/) (2022)
- [Recovering 3D Human Mesh from Monocular Images: A Survey](https://mlsanity.com/paper/2203.01923/) (2022)
- [Interacting Attention Graph for Single Image Two-Hand Reconstruction](https://mlsanity.com/paper/2203.09364/) (2022)
- [Hi4D: 4D Instance Segmentation of Close Human Interaction](https://mlsanity.com/paper/2303.15380/) (2023)
- [UDE: A Unified Driving Engine for Human Motion Generation](https://mlsanity.com/paper/2211.16016/) (2022)
- [Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives](https://mlsanity.com/paper/2403.10518/) (2024)
- [PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations](https://mlsanity.com/paper/2404.04421/) (2024)
- [Learning Implicit Templates for Point-Based Clothed Human Modeling](https://mlsanity.com/paper/2207.06955/) (2022)
- [ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction](https://mlsanity.com/paper/2303.05938/) (2023)
- [PACE: Human and Camera Motion Estimation from in-the-wild Videos](https://mlsanity.com/paper/2310.13768/) (2023)
- [Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera](https://mlsanity.com/paper/2401.00847/) (2024)
- [Animatable and Relightable Gaussians for High-fidelity Human Avatar Modeling](https://mlsanity.com/paper/2311.16096/) (2023)
- [FLAG3D: A 3D Fitness Activity Dataset with Language Instruction](https://mlsanity.com/paper/2212.04638/) (2022)
- [Neural Capture of Animatable 3D Human from Monocular Video](https://mlsanity.com/paper/2208.08728/) (2022)
- [Single-to-Dual-View Adaptation for Egocentric 3D Hand Pose Estimation](https://mlsanity.com/paper/2403.04381/) (2024)
- [MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors](https://mlsanity.com/paper/2403.17610/) (2024)
- [Delving Deep into Pixel Alignment Feature for Accurate Multi-view Human Mesh Recovery](https://mlsanity.com/paper/2301.06020/) (2023)
- [Progressive Multi-view Human Mesh Recovery with Self-Supervision](https://mlsanity.com/paper/2212.05223/) (2022)
- [ProxyCap: Real-time Monocular Full-body Capture in World Space via Human-Centric Proxy-to-Motion Learning](https://mlsanity.com/paper/2307.01200/) (2023)
- [HumanRAM: Feed-forward Human Reconstruction and Animation Model using Transformers](https://mlsanity.com/paper/2506.03118/) (2025)
- [Spatial-Temporal Parallel Transformer for Arm-Hand Dynamic Estimation](https://mlsanity.com/paper/2203.16202/) (2022)
- [Implicit 3D Human Mesh Recovery using Consistency with Pose and Shape from Unseen-view](https://mlsanity.com/paper/2306.17651/) (2023)
- [SmartPortraits: Depth Powered Handheld Smartphone Dataset of Human Portraits for State Estimation, Reconstruction and Synthesis](https://mlsanity.com/paper/2204.10211/) (2022)
- [AvatarPose: Avatar-guided 3D Pose Estimation of Close Human Interaction from Sparse Multi-view Videos](https://mlsanity.com/paper/2408.02110/) (2024)
- [LWA-HAND: Lightweight Attention Hand for Interacting Hand Reconstruction](https://mlsanity.com/paper/2208.09815/) (2022)
- [Simultaneously Recovering Multi-Person Meshes and Multi-View Cameras with Human Semantics](https://mlsanity.com/paper/2412.18785/) (2024)
- [Learning Context-Adaptive Motion Priors for Masked Motion Diffusion Models with Efficient Kinematic Attention Aggregation](https://mlsanity.com/paper/2603.07697/) (2026)

## Abstract (arXiv)

Multi-person total motion capture is extremely challenging when it comes to handle severe occlusions, different reconstruction granularities from body to face and hands, drastically changing observation scales and fast body movements. To overcome these challenges above, we contribute a lightweight total motion capture system for multi-person interactive scenarios using only sparse multi-view cameras. By contributing a novel hand and face bootstrapping algorithm, our method is capable of efficient localization and accurate association of the hands and faces even on severe occluded occasions. We leverage both pose regression and keypoints detection methods and further propose a unified two-stage parametric fitting method for achieving pixel-aligned accuracy. Moreover, for extremely self-occluded poses and close interactions, a novel feedback mechanism is proposed to propagate the pixel-aligned reconstructions into the next frame for more accurate association. Overall, we propose the first light-weight total capture system and achieves fast, robust and accurate multi-person total motion capture performance. The results and experiments show that our method achieves more accurate results than existing methods under sparse-view setups.