# NF3DM: Combining Neural Fields and Deformation Models for 3D Non-Rigid Motion Reconstruction

arXiv: https://arxiv.org/abs/2412.08511 · HTML: https://mlsanity.com/paper/2412.08511/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 67 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Neuralangelo: High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2306.03092/) (2023)
- [Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion](https://mlsanity.com/paper/2003.01456/) (2020)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018)
- [Sapiens: Foundation for Human Vision Models](https://mlsanity.com/paper/2408.12569/) (2024)
- [Learning to Reconstruct People in Clothing from a Single RGB Camera](https://mlsanity.com/paper/1903.05885/) (2019)
- [Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction](https://mlsanity.com/paper/2007.11432/) (2020)
- [4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface](https://mlsanity.com/paper/2105.01905/) (2021)
- [Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction](https://mlsanity.com/paper/2006.08072/) (2020)
- [Detailed Human Avatars from Monocular Video](https://mlsanity.com/paper/1808.01338/) (2018)
- [DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor](https://mlsanity.com/paper/1804.06023/) (2018)
- [NPMs: Neural Parametric Models for 3D Deformable Shapes](https://mlsanity.com/paper/2104.00702/) (2021)
- [Capturing and Animation of Body and Clothing from Monocular Video](https://mlsanity.com/paper/2210.01868/) (2022)
- [NeuroMorph: Unsupervised Shape Interpolation and Correspondence in One Go](https://mlsanity.com/paper/2106.09431/) (2021)
- [VolumeDeform: Real-time Volumetric Non-rigid Reconstruction](https://mlsanity.com/paper/1603.08161/) (2016)
- [Monocular Real-Time Volumetric Performance Capture](https://mlsanity.com/paper/2007.13988/) (2020)
- [Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction](https://mlsanity.com/paper/2012.01451/) (2020)
- [CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphism](https://mlsanity.com/paper/2203.16529/) (2022)
- [Dynamic Surface Function Networks for Clothed Human Bodies](https://mlsanity.com/paper/2104.03978/) (2021)

## Built on by (most cited first, 1 of 1)

- [Cepheid: Decoupling Scale from Structure for Cross-Family Motion Completion](https://mlsanity.com/paper/2610.18871/) (2026)

## Abstract (arXiv)

We introduce a novel, data-driven approach for reconstructing temporally coherent 3D motion from unstructured and potentially partial observations of non-rigidly deforming shapes. Our goal is to achieve high-fidelity motion reconstructions for shapes that undergo near-isometric deformations, such as humans wearing loose clothing. The key novelty of our work lies in its ability to combine implicit shape representations with explicit mesh-based deformation models, enabling detailed and temporally coherent motion reconstructions without relying on parametric shape models or decoupling shape and motion. Each frame is represented as a neural field decoded from a feature space where observations over time are fused, hence preserving geometric details present in the input data. Temporal coherence is enforced with a near-isometric deformation constraint between adjacent frames that applies to the underlying surface in the neural field. Our method outperforms state-of-the-art approaches, as demonstrated by its application to human and animal motion sequences reconstructed from monocular depth videos.