# HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances

arXiv: https://arxiv.org/abs/2210.05665 · HTML: https://mlsanity.com/paper/2210.05665/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 94 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 30 of 50)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018)
- [Learning to Estimate 3D Human Pose and Shape from a Single Color Image](https://mlsanity.com/paper/1805.04092/) (2018)
- [Unite the People: Closing the Loop Between 3D and 2D Human Representations](https://mlsanity.com/paper/1701.02468/) (2017)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [DeepHuman: 3D Human Reconstruction from a Single Image](https://mlsanity.com/paper/1903.06473/) (2019)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Monocular Total Capture: Posing Face, Body, and Hands in the Wild](https://mlsanity.com/paper/1812.01598/) (2018)
- [DeepCap: Monocular Human Performance Capture Using Weak Supervision](https://mlsanity.com/paper/2003.08325/) (2020)
- [SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization](https://mlsanity.com/paper/1912.07109/) (2019)
- [GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB](https://mlsanity.com/paper/1712.01057/) (2017)
- [SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video](https://mlsanity.com/paper/2201.12792/) (2022)
- [BodyNet: Volumetric Inference of 3D Human Body Shapes](https://mlsanity.com/paper/1804.04875/) (2018)
- [Collaborative Regression of Expressive Bodies using Moderation](https://mlsanity.com/paper/2105.05301/) (2021)
- [Real-time Deep Dynamic Characters](https://mlsanity.com/paper/2105.01794/) (2021)
- [Compositional Human Pose Regression](https://mlsanity.com/paper/1704.00159/) (2017)
- [End-to-end Hand Mesh Recovery from a Monocular RGB Image](https://mlsanity.com/paper/1902.09305/) (2019)

## Built on by (most cited first, 12 of 12)

- [InstantAvatar: Learning Avatars from Monocular Video in 60 Seconds](https://mlsanity.com/paper/2212.10550/) (2022)
- [ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering](https://mlsanity.com/paper/2312.05941/) (2023)
- [HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian Splatting](https://mlsanity.com/paper/2312.03461/) (2023)
- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)
- [Drivable Avatar Clothing: Faithful Full-Body Telepresence with Dynamic Clothing Driven by Sparse RGB-D Input](https://mlsanity.com/paper/2310.05917/) (2023)
- [GarVerseLOD: High-Fidelity 3D Garment Reconstruction from a Single In-the-Wild Image using a Dataset with Levels of Details](https://mlsanity.com/paper/2411.03047/) (2024)
- [DressRecon: Freeform 4D Human Reconstruction from Monocular Video](https://mlsanity.com/paper/2409.20563/) (2024)
- [Relightable Neural Actor with Intrinsic Decomposition and Pose Control](https://mlsanity.com/paper/2312.11587/) (2023)
- [MetaCap: Meta-learning Priors from Multi-View Imagery for Sparse-view Human Performance Capture and Rendering](https://mlsanity.com/paper/2403.18820/) (2024)
- [Wild2Avatar: Rendering Humans Behind Occlusions](https://mlsanity.com/paper/2401.00431/) (2023)
- [VINECS: Video-based Neural Character Skinning](https://mlsanity.com/paper/2307.00842/) (2023)
- [GaussianGAN: Real-Time Photorealistic controllable Human Avatars](https://mlsanity.com/paper/2509.01681/) (2025)

## Abstract (arXiv)

Monocular 3D human performance capture is indispensable for many applications in computer graphics and vision for enabling immersive experiences. However, detailed capture of humans requires tracking of multiple aspects, including the skeletal pose, the dynamic surface, which includes clothing, hand gestures as well as facial expressions. No existing monocular method allows joint tracking of all these components. To this end, we propose HiFECap, a new neural human performance capture approach, which simultaneously captures human pose, clothing, facial expression, and hands just from a single RGB video. We demonstrate that our proposed network architecture, the carefully designed training strategy, and the tight integration of parametric face and hand models to a template mesh enable the capture of all these individual aspects. Importantly, our method also captures high-frequency details, such as deforming wrinkles on the clothes, better than the previous works. Furthermore, we show that HiFECap outperforms the state-of-the-art human performance capture approaches qualitatively and quantitatively while for the first time capturing all aspects of the human.