# Dynamic Surface Function Networks for Clothed Human Bodies

arXiv: https://arxiv.org/abs/2104.03978 · HTML: https://mlsanity.com/paper/2104.03978/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 70 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion](https://mlsanity.com/paper/2003.01456/) (2020)
- [ARCH: Animatable Reconstruction of Clothed Humans](https://mlsanity.com/paper/2004.04572/) (2020)
- [Learning Shape Templates with Structured Implicit Functions](https://mlsanity.com/paper/1904.06447/) (2019)
- [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018)
- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)
- [SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks](https://mlsanity.com/paper/2104.03313/) (2021)
- [Learning to Reconstruct People in Clothing from a Single RGB Camera](https://mlsanity.com/paper/1903.05885/) (2019)
- [Tex2Shape: Detailed Full Human Body Geometry From a Single Image](https://mlsanity.com/paper/1904.08645/) (2019)
- [DeepCap: Monocular Human Performance Capture Using Weak Supervision](https://mlsanity.com/paper/2003.08325/) (2020)
- [SMPLicit: Topology-aware Generative Model for Clothed People](https://mlsanity.com/paper/2103.06871/) (2021)
- [NASA: Neural Articulated Shape Approximation](https://mlsanity.com/paper/1912.03207/) (2019)
- [Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction](https://mlsanity.com/paper/2007.11432/) (2020)
- [Detailed Human Avatars from Monocular Video](https://mlsanity.com/paper/1808.01338/) (2018)
- [NPMs: Neural Parametric Models for 3D Deformable Shapes](https://mlsanity.com/paper/2104.00702/) (2021)
- [Detailed, accurate, human shape estimation from clothed 3D scan sequences](https://mlsanity.com/paper/1703.04454/) (2017)
- [Graphonomy: Universal Human Parsing via Graph Transfer Learning](https://mlsanity.com/paper/1904.04536/) (2019)
- [PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations](https://mlsanity.com/paper/2008.01639/) (2020)
- [VolumeDeform: Real-time Volumetric Non-rigid Reconstruction](https://mlsanity.com/paper/1603.08161/) (2016)
- [Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction](https://mlsanity.com/paper/2012.01451/) (2020)
- [MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video](https://mlsanity.com/paper/2009.10711/) (2020)
- [Estimation of Human Body Shape and Posture Under Clothing](https://mlsanity.com/paper/1312.4967/) (2013)

## Built on by (most cited first, 30 of 44)

- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Neural Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2112.01554/) (2021)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [ARAH: Animatable Volume Rendering of Articulated Human SDFs](https://mlsanity.com/paper/2210.10036/) (2022)
- [Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition](https://mlsanity.com/paper/2302.11566/) (2023)
- [Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars](https://mlsanity.com/paper/2211.11208/) (2022)
- [Generalizable Patch-Based Neural Rendering](https://mlsanity.com/paper/2207.10662/) (2022)
- [The Power of Points for Modeling Humans in Clothing](https://mlsanity.com/paper/2109.01137/) (2021)
- [AvatarReX: Real-time Expressive Full-body Avatars](https://mlsanity.com/paper/2305.04789/) (2023)
- [MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images](https://mlsanity.com/paper/2106.11944/) (2021)
- [HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian Splatting](https://mlsanity.com/paper/2312.03461/) (2023)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars](https://mlsanity.com/paper/2205.08535/) (2022)
- [Learning Implicit Templates for Point-Based Clothed Human Modeling](https://mlsanity.com/paper/2207.06955/) (2022)
- [PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling](https://mlsanity.com/paper/2304.13006/) (2023)
- [PINA: Learning a Personalized Implicit Neural Avatar from a Single RGB-D Video Sequence](https://mlsanity.com/paper/2203.01754/) (2022)
- [NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions](https://mlsanity.com/paper/2202.12825/) (2022)
- [CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshes](https://mlsanity.com/paper/2206.04382/) (2022)
- [DANBO: Disentangled Articulated Neural Body Representations via Graph Neural Networks](https://mlsanity.com/paper/2205.01666/) (2022)
- [MeshAvatar: Learning High-quality Triangular Human Avatars from Multi-view Videos](https://mlsanity.com/paper/2407.08414/) (2024)
- [HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars](https://mlsanity.com/paper/2112.10203/) (2021)
- [Neural Point-based Shape Modeling of Humans in Challenging Clothing](https://mlsanity.com/paper/2209.06814/) (2022)
- [Neural Shape Deformation Priors](https://mlsanity.com/paper/2210.05616/) (2022)
- [Animatable and Relightable Gaussians for High-fidelity Human Avatar Modeling](https://mlsanity.com/paper/2311.16096/) (2023)
- [AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture](https://mlsanity.com/paper/2207.02031/) (2022)
- [PERGAMO: Personalized 3D Garments from Monocular Video](https://mlsanity.com/paper/2210.15040/) (2022)
- [LayGA: Layered Gaussian Avatars for Animatable Clothing Transfer](https://mlsanity.com/paper/2405.07319/) (2024)
- [CloSET: Modeling Clothed Humans on Continuous Surface with Explicit Template Decomposition](https://mlsanity.com/paper/2304.03167/) (2023)
- [CaPhy: Capturing Physical Properties for Animatable Human Avatars](https://mlsanity.com/paper/2308.05925/) (2023)
- [Dressing Avatars: Deep Photorealistic Appearance for Physically Simulated Clothing](https://mlsanity.com/paper/2206.15470/) (2022)

## Abstract (arXiv)

We present a novel method for temporal coherent reconstruction and tracking of clothed humans. Given a monocular RGB-D sequence, we learn a person-specific body model which is based on a dynamic surface function network. To this end, we explicitly model the surface of the person using a multi-layer perceptron (MLP) which is embedded into the canonical space of the SMPL body model. With classical forward rendering, the represented surface can be rasterized using the topology of a template mesh. For each surface point of the template mesh, the MLP is evaluated to predict the actual surface location. To handle pose-dependent deformations, the MLP is conditioned on the SMPL pose parameters. We show that this surface representation as well as the pose parameters can be learned in a self-supervised fashion using the principle of analysis-by-synthesis and differentiable rasterization. As a result, we are able to reconstruct a temporally coherent mesh sequence from the input data. The underlying surface representation can be used to synthesize new animations of the reconstructed person including pose-dependent deformations.