# Unbiased 4D: Monocular 4D Reconstruction with a Neural Deformation Model

arXiv: https://arxiv.org/abs/2206.08368 · HTML: https://mlsanity.com/paper/2206.08368/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 64 references that are in the map.
- First posted on arXiv 16 Jun 2022.

## Builds on (most cited first, 30 of 37)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [SAL: Sign Agnostic Learning of Shapes from Raw Data](https://mlsanity.com/paper/1911.10414/) (2019)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies](https://mlsanity.com/paper/2105.02872/) (2021)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes](https://mlsanity.com/paper/2104.06935/) (2021)
- [VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera](https://mlsanity.com/paper/1705.01583/) (2017)
- [LASR: Learning Articulated Shape Reconstruction from a Monocular Video](https://mlsanity.com/paper/2105.02976/) (2021)
- [Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control](https://mlsanity.com/paper/2106.02019/) (2021)
- [LiveCap: Real-time Human Performance Capture from Monocular Video](https://mlsanity.com/paper/1810.02648/) (2018)
- [C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion](https://mlsanity.com/paper/1909.02533/) (2019)

## Built on by (most cited first, 25 of 25)

- [NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction](https://mlsanity.com/paper/2212.05231/) (2022)
- [Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos](https://mlsanity.com/paper/2406.18717/) (2024)
- [Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction](https://mlsanity.com/paper/2312.01196/) (2023)
- [DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation](https://mlsanity.com/paper/2410.06756/) (2024)
- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Dynamic Scenes](https://mlsanity.com/paper/2404.12379/) (2024)
- [Decaf: Monocular Deformation Capture for Face and Hand Interactions](https://mlsanity.com/paper/2309.16670/) (2023)
- [7DGS: Unified Spatial-Temporal-Angular Gaussian Splatting](https://mlsanity.com/paper/2503.07946/) (2025)
- [DaReNeRF: Direction-aware Representation for Dynamic Scenes](https://mlsanity.com/paper/2403.02265/) (2024)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes](https://mlsanity.com/paper/2308.08258/) (2023)
- [MetaCap: Meta-learning Priors from Multi-View Imagery for Sparse-view Human Performance Capture and Rendering](https://mlsanity.com/paper/2403.18820/) (2024)
- [BLiRF: Bandlimited Radiance Fields for Dynamic Scene Modeling](https://mlsanity.com/paper/2302.13543/) (2023)
- [CanFields: Consolidating Diffeomorphic Flows for Non-Rigid 4D Interpolation from Arbitrary-Length Sequences](https://mlsanity.com/paper/2406.18582/) (2024)
- [Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules](https://mlsanity.com/paper/2412.04457/) (2024)
- [Adaptive and Temporally Consistent Gaussian Surfels for Multi-view Dynamic Reconstruction](https://mlsanity.com/paper/2411.06602/) (2024)
- [Fast Non-Rigid Radiance Fields from Monocularized Data](https://mlsanity.com/paper/2212.01368/) (2022)
- [Thin-Shell-SfT: Fine-Grained Monocular Non-rigid 3D Surface Tracking with Neural Deformation Fields](https://mlsanity.com/paper/2503.19976/) (2025)
- [Ev4DGS: Novel-view Rendering of Non-Rigid Objects from Monocular Event Streams](https://mlsanity.com/paper/2510.11717/) (2025)
- [Recovering Dynamic 3D Sketches from Videos](https://mlsanity.com/paper/2503.20321/) (2025)
- [A Probability-guided Sampler for Neural Implicit Surface Rendering](https://mlsanity.com/paper/2506.08619/) (2025)
- [Tessellation GS: Neural Mesh Gaussians for Robust Monocular Reconstruction of Dynamic Objects](https://mlsanity.com/paper/2512.07381/) (2025)
- [DaRePlane: Direction-aware Representations for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2410.14169/) (2024)
- [D-Prism: Differentiable Primitives for Structured Dynamic Modeling](https://mlsanity.com/paper/2604.17082/) (2026)

## Abstract (arXiv)

Capturing general deforming scenes from monocular RGB video is crucial for many computer graphics and vision applications. However, current approaches suffer from drawbacks such as struggling with large scene deformations, inaccurate shape completion or requiring 2D point tracks. In contrast, our method, Ub4D, handles large deformations, performs shape completion in occluded regions, and can operate on monocular RGB videos directly by using differentiable volume rendering. This technique includes three new in the context of non-rigid 3D reconstruction components, i.e., 1) A coordinate-based and implicit neural representation for non-rigid scenes, which in conjunction with differentiable volume rendering enables an unbiased reconstruction of dynamic scenes, 2) a proof that extends the unbiased formulation of volume rendering to dynamic scenes, and 3) a novel dynamic scene flow loss, which enables the reconstruction of larger deformations by leveraging the coarse estimates of other methods. Results on our new dataset, which will be made publicly available, demonstrate a clear improvement over the state of the art in terms of surface reconstruction accuracy and robustness to large deformations.