# Spatiotemporal Bundle Adjustment for Dynamic 3D Human Reconstruction in the Wild

arXiv: https://arxiv.org/abs/2007.12806 · HTML: https://mlsanity.com/paper/2007.12806/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2020.

## Built on by (most cited first, 11 of 11)

- [BANMo: Building Animatable 3D Neural Models from Many Casual Videos](https://mlsanity.com/paper/2112.12761/) (2021)
- [Consistent4D: Consistent 360deg Dynamic Object Generation from Monocular Video](https://mlsanity.com/paper/2311.02848/) (2023)
- [EgoHumans: An Egocentric 3D Multi-Human Benchmark](https://mlsanity.com/paper/2305.16487/) (2023)
- [Reconstructing Animatable Categories from Videos](https://mlsanity.com/paper/2305.06351/) (2023)
- [PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos](https://mlsanity.com/paper/2404.04430/) (2024)
- [MoDA: Modeling Deformable 3D Objects from Casual Videos](https://mlsanity.com/paper/2304.08279/) (2023)
- [CAMM: Building Category-Agnostic and Animatable 3D Models from Monocular Videos](https://mlsanity.com/paper/2304.06937/) (2023)
- [Diffusion-based 3D Hand Motion Recovery with Intuitive Physics](https://mlsanity.com/paper/2508.01835/) (2025)
- [Kineo: Calibration-Free Metric Motion Capture From Sparse RGB Cameras](https://mlsanity.com/paper/2510.24464/) (2025)
- [Spatiotemporal Multi-Camera Calibration using Freely Moving People](https://mlsanity.com/paper/2502.12546/) (2025)
- [3D Trajectory Reconstruction of Moving Points Based on Asynchronous Cameras](https://mlsanity.com/paper/2506.00541/) (2025)

## Abstract (arXiv)

Bundle adjustment jointly optimizes camera intrinsics and extrinsics and 3D point triangulation to reconstruct a static scene. The triangulation constraint, however, is invalid for moving points captured in multiple unsynchronized videos and bundle adjustment is not designed to estimate the temporal alignment between cameras. We present a spatiotemporal bundle adjustment framework that jointly optimizes four coupled sub-problems: estimating camera intrinsics and extrinsics, triangulating static 3D points, as well as sub-frame temporal alignment between cameras and computing 3D trajectories of dynamic points. Key to our joint optimization is the careful integration of physics-based motion priors within the reconstruction pipeline, validated on a large motion capture corpus of human subjects. We devise an incremental reconstruction and alignment algorithm to strictly enforce the motion prior during the spatiotemporal bundle adjustment. This algorithm is further made more efficient by a divide and conquer scheme while still maintaining high accuracy. We apply this algorithm to reconstruct 3D motion trajectories of human bodies in dynamic events captured by multiple uncalibrated and unsynchronized video cameras in the wild. To make the reconstruction visually more interpretable, we fit a statistical 3D human body model to the asynchronous video streams.Compared to the baseline, the fitting significantly benefits from the proposed spatiotemporal bundle adjustment procedure. Because the videos are aligned with sub-frame precision, we reconstruct 3D motion at much higher temporal resolution than the input videos.