# Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction

arXiv: https://arxiv.org/abs/2012.01451 · HTML: https://mlsanity.com/paper/2012.01451/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2020.

## Built on by (most cited first, 30 of 67)

- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [ICON: Implicit Clothed humans Obtained from Normals](https://mlsanity.com/paper/2112.09127/) (2021)
- [SpatialTracker: Tracking Any 2D Pixels in 3D Space](https://mlsanity.com/paper/2404.04319/) (2024)
- [Ditto: Building Digital Twins of Articulated Objects from Interaction](https://mlsanity.com/paper/2202.08227/) (2022)
- [Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition](https://mlsanity.com/paper/2302.11566/) (2023)
- [Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation](https://mlsanity.com/paper/2201.07786/) (2022)
- [Structured Local Radiance Fields for Human Avatar Modeling](https://mlsanity.com/paper/2203.14478/) (2022)
- [Neural-GIF: Neural Generalized Implicit Functions for Animating People in Clothing](https://mlsanity.com/paper/2108.08807/) (2021)
- [NPMs: Neural Parametric Models for 3D Deformable Shapes](https://mlsanity.com/paper/2104.00702/) (2021)
- [The Power of Points for Modeling Humans in Clothing](https://mlsanity.com/paper/2109.01137/) (2021)
- [SHERF: Generalizable Human NeRF from a Single Image](https://mlsanity.com/paper/2303.12791/) (2023)
- [HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian Splatting](https://mlsanity.com/paper/2312.03461/) (2023)
- [Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations](https://mlsanity.com/paper/2204.08839/) (2022)
- [A Survey of Non-Rigid 3D Registration](https://mlsanity.com/paper/2203.07858/) (2022)
- [CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphism](https://mlsanity.com/paper/2203.16529/) (2022)
- [Self-supervised Neural Articulated Shape and Appearance Models](https://mlsanity.com/paper/2205.08525/) (2022)
- [Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects](https://mlsanity.com/paper/2112.11347/) (2021)
- [PINA: Learning a Personalized Implicit Neural Avatar from a Single RGB-D Video Sequence](https://mlsanity.com/paper/2203.01754/) (2022)
- [Dynamic Surface Function Networks for Clothed Human Bodies](https://mlsanity.com/paper/2104.03978/) (2021)
- [NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions](https://mlsanity.com/paper/2202.12825/) (2022)
- [CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshes](https://mlsanity.com/paper/2206.04382/) (2022)
- [Predicting Loose-Fitting Garment Deformations Using Bone-Driven Motion Networks](https://mlsanity.com/paper/2205.01355/) (2022)
- [Multiway Non-rigid Point Cloud Registration via Learned Functional Map Synchronization](https://mlsanity.com/paper/2111.12878/) (2021)
- [SPAMs: Structured Implicit Parametric Models](https://mlsanity.com/paper/2201.08141/) (2022)
- [Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking](https://mlsanity.com/paper/2401.06614/) (2024)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [OcclusionFusion: Occlusion-aware Motion Estimation for Real-time Dynamic 3D Reconstruction](https://mlsanity.com/paper/2203.07977/) (2022)
- [Neural Shape Deformation Priors](https://mlsanity.com/paper/2210.05616/) (2022)
- [Human Performance Capture from Monocular Video in the Wild](https://mlsanity.com/paper/2111.14672/) (2021)
- [NAP: Neural 3D Articulation Prior](https://mlsanity.com/paper/2305.16315/) (2023)

## Abstract (arXiv)

We introduce Neural Deformation Graphs for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects. Specifically, we implicitly model a deformation graph via a deep neural network. This neural deformation graph does not rely on any object-specific structure and, thus, can be applied to general non-rigid deformation tracking. Our method globally optimizes this neural graph on a given sequence of depth camera observations of a non-rigidly moving object. Based on explicit viewpoint consistency as well as inter-frame graph and surface consistency constraints, the underlying network is trained in a self-supervised fashion. We additionally optimize for the geometry of the object with an implicit deformable multi-MLP shape representation. Our approach does not assume sequential input data, thus enabling robust tracking of fast motions or even temporally disconnected recordings. Our experiments demonstrate that our Neural Deformation Graphs outperform state-of-the-art non-rigid reconstruction approaches both qualitatively and quantitatively, with 64% improved reconstruction and 62% improved deformation tracking performance.