# Multi-body Non-rigid Structure-from-Motion

arXiv: https://arxiv.org/abs/1607.04515 · HTML: https://mlsanity.com/paper/1607.04515/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2016.

## Built on by (most cited first, 14 of 14)

- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)
- [Organic Priors in Non-Rigid Structure from Motion](https://mlsanity.com/paper/2207.06262/) (2022)
- [PAUL: Procrustean Autoencoder for Unsupervised Lifting](https://mlsanity.com/paper/2103.16773/) (2021)
- [Deep Non-rigid Structure-from-Motion: A Sequence-to-Sequence Translation Perspective](https://mlsanity.com/paper/2204.04730/) (2022)
- [RoDyGS: Robust Dynamic Gaussian Splatting for Casual Videos](https://mlsanity.com/paper/2412.03077/) (2024)
- [View Birdification in the Crowd: Ground-Plane Localization from Perceived Movements](https://mlsanity.com/paper/2111.05060/) (2021)
- [Uncertainty-Driven Dense Two-View Structure from Motion](https://mlsanity.com/paper/2302.00523/) (2023)
- [Learning Better Keypoints for Multi-Object 6DoF Pose Estimation](https://mlsanity.com/paper/2308.07827/) (2023)
- [High Fidelity 3D Reconstructions with Limited Physical Views](https://mlsanity.com/paper/2110.11599/) (2021)
- [Time-Archival Camera Virtualization for Sports and Visual Performances](https://mlsanity.com/paper/2602.15181/) (2026)
- [Reconstructing Small 3D Objects in front of a Textured Background](https://mlsanity.com/paper/2105.11352/) (2021)
- [Camera Movement Estimation and Path Correction using the Combination of Modified A-SIFT and Stereo System for 3D Modelling](https://mlsanity.com/paper/2503.17668/) (2025)
- [A Dataset and Evaluation for Complex 4D Markerless Human Motion Capture](https://mlsanity.com/paper/2604.12765/) (2026)
- [MoSE3: Learning World-Space SE(3) at Every Pixel](https://mlsanity.com/paper/2610.03716/) (2026)

## Abstract (arXiv)

Conventional structure-from-motion (SFM) research is primarily concerned with the 3D reconstruction of a single, rigidly moving object seen by a static camera, or a static and rigid scene observed by a moving camera --in both cases there are only one relative rigid motion involved. Recent progress have extended SFM to the areas of {multi-body SFM} (where there are {multiple rigid} relative motions in the scene), as well as {non-rigid SFM} (where there is a single non-rigid, deformable object or scene). Along this line of thinking, there is apparently a missing gap of "multi-body non-rigid SFM", in which the task would be to jointly reconstruct and segment multiple 3D structures of the multiple, non-rigid objects or deformable scenes from images. Such a multi-body non-rigid scenario is common in reality (e.g. two persons shaking hands, multi-person social event), and how to solve it represents a natural {next-step} in SFM research. By leveraging recent results of subspace clustering, this paper proposes, for the first time, an effective framework for multi-body NRSFM, which simultaneously reconstructs and segments each 3D trajectory into their respective low-dimensional subspace. Under our formulation, 3D trajectories for each non-rigid structure can be well approximated with a sparse affine combination of other 3D trajectories from the same structure (self-expressiveness). We solve the resultant optimization with the alternating direction method of multipliers (ADMM). We demonstrate the efficacy of the proposed framework through extensive experiments on both synthetic and real data sequences. Our method clearly outperforms other alternative methods, such as first clustering the 2D feature tracks to groups and then doing non-rigid reconstruction in each group or first conducting 3D reconstruction by using single subspace assumption and then clustering the 3D trajectories into groups.