# Deep Non-Rigid Structure from Motion

arXiv: https://arxiv.org/abs/1908.00052 · HTML: https://mlsanity.com/paper/1908.00052/

## Key facts

- Cited by 38 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 30 Jul 2019.

## Built on by (most cited first, 30 of 38)

- [BANMo: Building Animatable 3D Neural Models from Many Casual Videos](https://mlsanity.com/paper/2112.12761/) (2021)
- [LASR: Learning Articulated Shape Reconstruction from a Monocular Video](https://mlsanity.com/paper/2105.02976/) (2021)
- [Shelf-Supervised Mesh Prediction in the Wild](https://mlsanity.com/paper/2102.06195/) (2021)
- [Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera](https://mlsanity.com/paper/2206.15258/) (2022)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [CASA: Category-agnostic Skeletal Animal Reconstruction](https://mlsanity.com/paper/2211.03568/) (2022)
- [Pre-train, Self-train, Distill: A simple recipe for Supersizing 3D Reconstruction](https://mlsanity.com/paper/2204.03642/) (2022)
- [Uncertainty-Aware Adaptation for Self-Supervised 3D Human Pose Estimation](https://mlsanity.com/paper/2203.15293/) (2022)
- [Virtual Correspondence: Humans as a Cue for Extreme-View Geometry](https://mlsanity.com/paper/2206.08365/) (2022)
- [Unbiased 4D: Monocular 4D Reconstruction with a Neural Deformation Model](https://mlsanity.com/paper/2206.08368/) (2022)
- [To The Point: Correspondence-driven monocular 3D category reconstruction](https://mlsanity.com/paper/2106.05662/) (2021)
- [BAGS: Building Animatable Gaussian Splatting from a Monocular Video with Diffusion Priors](https://mlsanity.com/paper/2403.11427/) (2024)
- [Fast Encoder-Based 3D from Casual Videos via Point Track Processing](https://mlsanity.com/paper/2404.07097/) (2024)
- [OMAD: Object Model with Articulated Deformations for Pose Estimation and Retrieval](https://mlsanity.com/paper/2112.07334/) (2021)
- [MovingParts: Motion-based 3D Part Discovery in Dynamic Radiance Field](https://mlsanity.com/paper/2303.05703/) (2023)
- [DreaMo: Articulated 3D Reconstruction From A Single Casual Video](https://mlsanity.com/paper/2312.02617/) (2023)
- [CAMM: Building Category-Agnostic and Animatable 3D Models from Monocular Videos](https://mlsanity.com/paper/2304.06937/) (2023)
- [PAUL: Procrustean Autoencoder for Unsupervised Lifting](https://mlsanity.com/paper/2103.16773/) (2021)
- [Root Pose Decomposition Towards Generic Non-rigid 3D Reconstruction with Monocular Videos](https://mlsanity.com/paper/2308.10089/) (2023)
- [3D-LFM: Lifting Foundation Model](https://mlsanity.com/paper/2312.11894/) (2023)
- [Non-Local Latent Relation Distillation for Self-Adaptive 3D Human Pose Estimation](https://mlsanity.com/paper/2204.01971/) (2022)
- [DensePose 3D: Lifting Canonical Surface Maps of Articulated Objects to the Third Dimension](https://mlsanity.com/paper/2109.00033/) (2021)
- [The Drunkard's Odometry: Estimating Camera Motion in Deforming Scenes](https://mlsanity.com/paper/2306.16917/) (2023)
- [Camera-Independent Single Image Depth Estimation from Defocus Blur](https://mlsanity.com/paper/2311.13045/) (2023)
- [4DPV: 4D Pet from Videos by Coarse-to-Fine Non-Rigid Radiance Fields](https://mlsanity.com/paper/2411.10275/) (2024)
- [DynamicSurf: Dynamic Neural RGB-D Surface Reconstruction with an Optimizable Feature Grid](https://mlsanity.com/paper/2311.08159/) (2023)
- [GTT-Net: Learned Generalized Trajectory Triangulation](https://mlsanity.com/paper/2109.03408/) (2021)
- [High Fidelity 3D Reconstructions with Limited Physical Views](https://mlsanity.com/paper/2110.11599/) (2021)
- [Learning Internal Representations of 3D Transformations from 2D Projected Inputs](https://mlsanity.com/paper/2303.17776/) (2023)
- [What is Wrong with End-to-End Learning for Phase Retrieval?](https://mlsanity.com/paper/2403.15448/) (2024)

## Abstract (arXiv)

Current non-rigid structure from motion (NRSfM) algorithms are mainly limited with respect to: (i) the number of images, and (ii) the type of shape variability they can handle. This has hampered the practical utility of NRSfM for many applications within vision. In this paper we propose a novel deep neural network to recover camera poses and 3D points solely from an ensemble of 2D image coordinates. The proposed neural network is mathematically interpretable as a multi-layer block sparse dictionary learning problem, and can handle problems of unprecedented scale and shape complexity. Extensive experiments demonstrate the impressive performance of our approach where we exhibit superior precision and robustness against all available state-of-the-art works in the order of magnitude. We further propose a quality measure (based on the network weights) which circumvents the need for 3D ground-truth to ascertain the confidence we have in the reconstruction.