# Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera

arXiv: https://arxiv.org/abs/2206.15258 · HTML: https://mlsanity.com/paper/2206.15258/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 76 references that are in the map.
- First posted on arXiv 30 Jun 2022.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Dynamic View Synthesis from Dynamic Monocular Video](https://mlsanity.com/paper/2105.06468/) (2021)
- [SAL: Sign Agnostic Learning of Shapes from Raw Data](https://mlsanity.com/paper/1911.10414/) (2019)
- [Invertible Residual Networks](https://mlsanity.com/paper/1811.00995/) (2018)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction](https://mlsanity.com/paper/2007.03858/) (2020)
- [HeadNeRF: A Real-time NeRF-based Parametric Head Model](https://mlsanity.com/paper/2112.05637/) (2021)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [Neural Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2112.01554/) (2021)
- [BANMo: Building Animatable 3D Neural Models from Many Casual Videos](https://mlsanity.com/paper/2112.12761/) (2021)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [DeepCap: Monocular Human Performance Capture Using Weak Supervision](https://mlsanity.com/paper/2003.08325/) (2020)
- [SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video](https://mlsanity.com/paper/2201.12792/) (2022)
- [Robust High-Resolution Video Matting with Temporal Guidance](https://mlsanity.com/paper/2108.11515/) (2021)
- [Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion](https://mlsanity.com/paper/2103.07941/) (2021)
- [LASR: Learning Articulated Shape Reconstruction from a Monocular Video](https://mlsanity.com/paper/2105.02976/) (2021)

## Built on by (most cited first, 30 of 57)

- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images](https://mlsanity.com/paper/2311.13398/) (2023)
- [Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models](https://mlsanity.com/paper/2312.13763/) (2023)
- [OmniRe: Omni Urban Scene Reconstruction](https://mlsanity.com/paper/2408.16760/) (2024)
- [HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian Splatting](https://mlsanity.com/paper/2312.03461/) (2023)
- [EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos](https://mlsanity.com/paper/2307.11307/) (2023)
- [NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions](https://mlsanity.com/paper/2212.07626/) (2022)
- [Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction](https://mlsanity.com/paper/2312.01196/) (2023)
- [SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstruction](https://mlsanity.com/paper/2409.11211/) (2024)
- [Grid4D: 4D Decomposed Hash Encoding for High-Fidelity Dynamic Gaussian Splatting](https://mlsanity.com/paper/2410.20815/) (2024)
- [Flow supervision for Deformable NeRF](https://mlsanity.com/paper/2303.16333/) (2023)
- [I2VControl-Camera: Precise Video Camera Control with Adjustable Motion Strength](https://mlsanity.com/paper/2411.06525/) (2024)
- [Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels](https://mlsanity.com/paper/2405.16822/) (2024)
- [Relightable and Animatable Neural Avatar from Sparse-View Video](https://mlsanity.com/paper/2308.07903/) (2023)
- [NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos](https://mlsanity.com/paper/2312.06398/) (2023)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [ResFields: Residual Neural Fields for Spatiotemporal Signals](https://mlsanity.com/paper/2309.03160/) (2023)
- [Instant-NVR: Instant Neural Volumetric Rendering for Human-object Interactions from Monocular RGBD Stream](https://mlsanity.com/paper/2304.03184/) (2023)
- [Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis](https://mlsanity.com/paper/2304.12317/) (2023)
- [NViST: In the Wild New View Synthesis from a Single Image with Transformers](https://mlsanity.com/paper/2312.08568/) (2023)
- [4DTAM: Non-Rigid Tracking and Mapping via Dynamic Surface Gaussians](https://mlsanity.com/paper/2505.22859/) (2025)
- [SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes](https://mlsanity.com/paper/2308.08258/) (2023)
- [MV-DeepSDF: Implicit Modeling with Multi-Sweep Point Clouds for 3D Vehicle Reconstruction in Autonomous Driving](https://mlsanity.com/paper/2309.16715/) (2023)
- [TRACE: Learning 3D Gaussian Physical Dynamics from Multi-view Videos](https://mlsanity.com/paper/2508.09811/) (2025)
- [Root Pose Decomposition Towards Generic Non-rigid 3D Reconstruction with Monocular Videos](https://mlsanity.com/paper/2308.10089/) (2023)
- [3D LiDAR Mapping in Dynamic Environments Using a 4D Implicit Neural Representation](https://mlsanity.com/paper/2405.03388/) (2024)
- [DASH: 4D Hash Encoding with Self-Supervised Decomposition for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2507.19141/) (2025)
- [UrbanGS: Semantic-Guided Gaussian Splatting for Urban Scene Reconstruction](https://mlsanity.com/paper/2412.03473/) (2024)
- [High-Quality Mesh Blendshape Generation from Face Videos via Neural Inverse Rendering](https://mlsanity.com/paper/2401.08398/) (2024)
- [Space-time 2D Gaussian Splatting for Accurate Surface Reconstruction under Complex Dynamic Scenes](https://mlsanity.com/paper/2409.18852/) (2024)

## Abstract (arXiv)

We propose Neural-DynamicReconstruction (NDR), a template-free method to recover high-fidelity geometry and motions of a dynamic scene from a monocular RGB-D camera. In NDR, we adopt the neural implicit function for surface representation and rendering such that the captured color and depth can be fully utilized to jointly optimize the surface and deformations. To represent and constrain the non-rigid deformations, we propose a novel neural invertible deforming network such that the cycle consistency between arbitrary two frames is automatically satisfied. Considering that the surface topology of dynamic scene might change over time, we employ a topology-aware strategy to construct the topology-variant correspondence for the fused frames. NDR also further refines the camera poses in a global optimization manner. Experiments on public datasets and our collected dataset demonstrate that NDR outperforms existing monocular dynamic reconstruction methods.