# Imposing Temporal Consistency on Deep Monocular Body Shape and Pose Estimation

arXiv: https://arxiv.org/abs/2202.03074 · HTML: https://mlsanity.com/paper/2202.03074/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 7 Feb 2022.

## Builds on (most cited first, 26 of 26)

- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop](https://mlsanity.com/paper/2103.16507/) (2021)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [Deep Learning-Based Human Pose Estimation: A Survey](https://mlsanity.com/paper/2012.13392/) (2020)
- [Unite the People: Closing the Loop Between 3D and 2D Human Representations](https://mlsanity.com/paper/1701.02468/) (2017)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [Monocular Total Capture: Posing Face, Body, and Hands in the Wild](https://mlsanity.com/paper/1812.01598/) (2018)
- [Monocular Expressive Body Regression through Body-Driven Attention](https://mlsanity.com/paper/2008.09062/) (2020)
- [Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild](https://mlsanity.com/paper/2007.15649/) (2020)
- [FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration](https://mlsanity.com/paper/2008.08324/) (2020)
- [EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera](https://mlsanity.com/paper/1908.11505/) (2019)
- [MoVi: A Large Multipurpose Motion and Video Dataset](https://mlsanity.com/paper/2003.01888/) (2020)
- [Monocular Real-time Full Body Capture with Inter-part Correlations](https://mlsanity.com/paper/2012.06087/) (2020)
- [3D Human Shape Reconstruction from a Polarization Image](https://mlsanity.com/paper/2007.09268/) (2020)
- [3D Human Pose, Shape and Texture from Low-Resolution Images and Videos](https://mlsanity.com/paper/2103.06498/) (2021)
- [ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References](https://mlsanity.com/paper/2103.06747/) (2021)
- [SparseFusion: Dynamic Human Avatar Modeling from Sparse RGBD Images](https://mlsanity.com/paper/2006.03630/) (2020)
- [Temporal Consistency Loss for High Resolution Textured and Clothed 3DHuman Reconstruction from Monocular Video](https://mlsanity.com/paper/2104.09259/) (2021)

## Built on by (most cited first, 2 of 2)

- [Programmable Motion Generation for Open-Set Motion Control Tasks](https://mlsanity.com/paper/2405.19283/) (2024)
- [Animatable Virtual Humans: Learning pose-dependent human representations in UV space for interactive performance synthesis](https://mlsanity.com/paper/2310.03615/) (2023)

## Abstract (arXiv)

Accurate and temporally consistent modeling of human bodies is essential for a wide range of applications, including character animation, understanding human social behavior and AR/VR interfaces. Capturing human motion accurately from a monocular image sequence is still challenging and the modeling quality is strongly influenced by the temporal consistency of the captured body motion. Our work presents an elegant solution for the integration of temporal constraints in the fitting process. This does not only increase temporal consistency but also robustness during the optimization. In detail, we derive parameters of a sequence of body models, representing shape and motion of a person, including jaw poses, facial expressions, and finger poses. We optimize these parameters over the complete image sequence, fitting one consistent body shape while imposing temporal consistency on the body motion, assuming linear body joint trajectories over a short time. Our approach enables the derivation of realistic 3D body models from image sequences, including facial expression and articulated hands. In extensive experiments, we show that our approach results in accurately estimated body shape and motion, also for challenging movements and poses. Further, we apply it to the special application of sign language analysis, where accurate and temporal consistent motion modelling is essential, and show that the approach is well-suited for this kind of application.