# MulayCap: Multi-layer Human Performance Capture Using A Monocular Video Camera

arXiv: https://arxiv.org/abs/2004.05815 · HTML: https://mlsanity.com/paper/2004.05815/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2020.

## Built on by (most cited first, 14 of 14)

- [Real-time Deep Dynamic Characters](https://mlsanity.com/paper/2105.01794/) (2021)
- [DressCode: Autoregressively Sewing and Generating Garments from Text Guidance](https://mlsanity.com/paper/2401.16465/) (2024)
- [TORE: Token Reduction for Efficient Human Mesh Recovery with Transformer](https://mlsanity.com/paper/2211.10705/) (2022)
- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)
- [ChatGarment: Garment Estimation, Generation and Editing via Large Language Models](https://mlsanity.com/paper/2412.17811/) (2024)
- [PERGAMO: Personalized 3D Garments from Monocular Video](https://mlsanity.com/paper/2210.15040/) (2022)
- [CaPhy: Capturing Physical Properties for Animatable Human Avatars](https://mlsanity.com/paper/2308.05925/) (2023)
- [ReLoo: Reconstructing Humans Dressed in Loose Garments from Monocular Video in the Wild](https://mlsanity.com/paper/2409.15269/) (2024)
- [Layered-Garment Net: Generating Multiple Implicit Garment Layers from a Single Image](https://mlsanity.com/paper/2211.11931/) (2022)
- [Neural-ABC: Neural Parametric Models for Articulated Body with Clothes](https://mlsanity.com/paper/2404.04673/) (2024)
- [Class-agnostic Reconstruction of Dynamic Objects from Videos](https://mlsanity.com/paper/2112.02091/) (2021)
- [The Drunkard's Odometry: Estimating Camera Motion in Deforming Scenes](https://mlsanity.com/paper/2306.16917/) (2023)
- [MOSS: Motion-based 3D Clothed Human Synthesis from Monocular Video](https://mlsanity.com/paper/2405.12806/) (2024)
- [GarmentWeaver: Schema-Aware Structured Synthesis for Multimodal Sewing Patterns](https://mlsanity.com/paper/2608.30550/) (2026)

## Abstract (arXiv)

We introduce MulayCap, a novel human performance capture method using a monocular video camera without the need for pre-scanning. The method uses "multi-layer" representations for geometry reconstruction and texture rendering, respectively. For geometry reconstruction, we decompose the clothed human into multiple geometry layers, namely a body mesh layer and a garment piece layer. The key technique behind is a Garment-from-Video (GfV) method for optimizing the garment shape and reconstructing the dynamic cloth to fit the input video sequence, based on a cloth simulation model which is effectively solved with gradient descent. For texture rendering, we decompose each input image frame into a shading layer and an albedo layer, and propose a method for fusing a fixed albedo map and solving for detailed garment geometry using the shading layer. Compared with existing single view human performance capture systems, our "multi-layer" approach bypasses the tedious and time consuming scanning step for obtaining a human specific mesh template. Experimental results demonstrate that MulayCap produces realistic rendering of dynamically changing details that has not been achieved in any previous monocular video camera systems. Benefiting from its fully semantic modeling, MulayCap can be applied to various important editing applications, such as cloth editing, re-targeting, relighting, and AR applications.