# Learned Vertex Descent: A New Direction for 3D Human Model Fitting

arXiv: https://arxiv.org/abs/2205.06254 · HTML: https://mlsanity.com/paper/2205.06254/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 88 references that are in the map.
- First posted on arXiv 12 May 2022.

## Builds on (most cited first, 30 of 64)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [Neural Unsigned Distance Fields for Implicit Function Learning](https://mlsanity.com/paper/2010.13938/) (2020)
- [ARCH: Animatable Reconstruction of Clothed Humans](https://mlsanity.com/paper/2004.04572/) (2020)
- [Multi-Garment Net: Learning to Dress 3D People from Images](https://mlsanity.com/paper/1908.06903/) (2019)
- [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018)
- [PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction](https://mlsanity.com/paper/2007.03858/) (2020)
- [Learning to Estimate 3D Human Pose and Shape from a Single Color Image](https://mlsanity.com/paper/1805.04092/) (2018)
- [SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes](https://mlsanity.com/paper/2104.03953/) (2021)
- [Unite the People: Closing the Loop Between 3D and 2D Human Representations](https://mlsanity.com/paper/1701.02468/) (2017)
- [Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation](https://mlsanity.com/paper/1808.05942/) (2018)
- [SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks](https://mlsanity.com/paper/2104.03313/) (2021)

## Built on by (most cited first, 30 of 37)

- [Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2407.07532/) (2024)
- [Visibility Aware Human-Object Interaction Tracking from Single RGB Camera](https://mlsanity.com/paper/2303.16479/) (2023)
- [3D Human Mesh Estimation from Virtual Markers](https://mlsanity.com/paper/2303.11726/) (2023)
- [Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation](https://mlsanity.com/paper/2312.07063/) (2023)
- [Object pop-up: Can we infer 3D objects and their poses from human interactions alone?](https://mlsanity.com/paper/2306.00777/) (2023)
- [Human3R: Everyone Everywhere All at Once](https://mlsanity.com/paper/2510.06219/) (2025)
- [SplatArmor: Articulated Gaussian splatting for animatable humans from monocular RGB videos](https://mlsanity.com/paper/2311.10812/) (2023)
- [Learning Clothing and Pose Invariant 3D Shape Representation for Long-Term Person Re-Identification](https://mlsanity.com/paper/2308.10658/) (2023)
- [3D-Aware Neural Body Fitting for Occlusion Robust 3D Human Pose Estimation](https://mlsanity.com/paper/2308.10123/) (2023)
- [InterTrack: Tracking Human Object Interaction without Object Templates](https://mlsanity.com/paper/2408.13953/) (2024)
- [Drivable Avatar Clothing: Faithful Full-Body Telepresence with Dynamic Clothing Driven by Sparse RGB-D Input](https://mlsanity.com/paper/2310.05917/) (2023)
- [NICP: Neural ICP for 3D Human Registration at Scale](https://mlsanity.com/paper/2312.14024/) (2023)
- [Structured 3D Features for Reconstructing Controllable Avatars](https://mlsanity.com/paper/2212.06820/) (2022)
- [MEGA: Masked Generative Autoencoder for Human Mesh Recovery](https://mlsanity.com/paper/2405.18839/) (2024)
- [VQ-HPS: Human Pose and Shape Estimation in a Vector-Quantized Latent Space](https://mlsanity.com/paper/2312.08291/) (2023)
- [Delving Deep into Pixel Alignment Feature for Accurate Multi-view Human Mesh Recovery](https://mlsanity.com/paper/2301.06020/) (2023)
- [ProxyCap: Real-time Monocular Full-body Capture in World Space via Human-Centric Proxy-to-Motion Learning](https://mlsanity.com/paper/2307.01200/) (2023)
- [MeshPose: Unifying DensePose and 3D Body Mesh reconstruction](https://mlsanity.com/paper/2406.10180/) (2024)
- [KBody: Towards general, robust, and aligned monocular whole-body estimation](https://mlsanity.com/paper/2304.11542/) (2023)
- [ShapeBoost: Boosting Human Shape Estimation with Part-Based Parameterization and Clothing-Preserving Augmentation](https://mlsanity.com/paper/2403.01345/) (2024)
- [Reconstructing 3D Human Pose from RGB-D Data with Occlusions](https://mlsanity.com/paper/2310.01228/) (2023)
- [Learning Canonical Embeddings for Unsupervised Shape Correspondence with Locally Linear Transformations](https://mlsanity.com/paper/2209.02152/) (2022)
- [Mesh Strikes Back: Fast and Efficient Human Reconstruction from RGB videos](https://mlsanity.com/paper/2303.08808/) (2023)
- [3D Kinematics Estimation from Video with a Biomechanical Model and Synthetic Training Data](https://mlsanity.com/paper/2402.13172/) (2024)
- [PHD: Personalized 3D Human Body Fitting with Point Diffusion](https://mlsanity.com/paper/2508.21257/) (2025)
- [ETCH-X: Robustify Expressive Body Fitting to Clothed Humans with Composable Datasets](https://mlsanity.com/paper/2604.08548/) (2026)
- [GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction](https://mlsanity.com/paper/2604.19624/) (2026)
- [Motion-DVAE: Unsupervised learning for fast human motion denoising](https://mlsanity.com/paper/2306.05846/) (2023)
- [SwimXYZ: A large-scale dataset of synthetic swimming motions and videos](https://mlsanity.com/paper/2310.04360/) (2023)
- [SPARE: Symmetrized Point-to-Plane Distance for Robust Non-Rigid 3D Registration](https://mlsanity.com/paper/2405.20188/) (2024)

## Abstract (arXiv)

We propose a novel optimization-based paradigm for 3D human model fitting on images and scans. In contrast to existing approaches that directly regress the parameters of a low-dimensional statistical body model (e.g. SMPL) from input images, we train an ensemble of per-vertex neural fields network. The network predicts, in a distributed manner, the vertex descent direction towards the ground truth, based on neural features extracted at the current vertex projection. At inference, we employ this network, dubbed LVD, within a gradient-descent optimization pipeline until its convergence, which typically occurs in a fraction of a second even when initializing all vertices into a single point. An exhaustive evaluation demonstrates that our approach is able to capture the underlying body of clothed people with very different body shapes, achieving a significant improvement compared to state-of-the-art. LVD is also applicable to 3D model fitting of humans and hands, for which we show a significant improvement to the SOTA with a much simpler and faster method.