# Reconstructing People, Places, and Cameras

arXiv: https://arxiv.org/abs/2412.17806 · HTML: https://mlsanity.com/paper/2412.17806/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 72 references that are in the map.
- First posted on arXiv 23 Dec 2024.

## Builds on (most cited first, 30 of 49)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion](https://mlsanity.com/paper/2410.03825/) (2024)
- [Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://mlsanity.com/paper/1903.07071/) (2019)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion](https://mlsanity.com/paper/2312.07531/) (2023)
- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [Capturing and Inferring Dense Full-Body Human-Scene Contact](https://mlsanity.com/paper/2206.09553/) (2022)
- [ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer](https://mlsanity.com/paper/2208.14201/) (2022)
- [MASt3R-SfM: a Fully-Integrated Solution for Unconstrained Structure-from-Motion](https://mlsanity.com/paper/2409.19152/) (2024)
- [GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras](https://mlsanity.com/paper/2112.01524/) (2021)
- [PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustment](https://mlsanity.com/paper/2306.15667/) (2023)
- [Putting People in their Place: Monocular Regression of 3D People in Depth](https://mlsanity.com/paper/2112.08274/) (2021)
- [Decoupling Human and Camera Motion from Videos in the Wild](https://mlsanity.com/paper/2302.12827/) (2023)
- [GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences](https://mlsanity.com/paper/1912.05524/) (2019)
- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [World-Grounded Human Motion Recovery via Gravity-View Coordinates](https://mlsanity.com/paper/2409.06662/) (2024)
- [Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters](https://mlsanity.com/paper/1904.00889/) (2019)
- [HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling](https://mlsanity.com/paper/2204.13686/) (2022)

## Built on by (most cited first, 30 of 31)

- [Visual Imitation Enables Contextual Humanoid Control](https://mlsanity.com/paper/2505.03729/) (2025)
- [Human3R: Everyone Everywhere All at Once](https://mlsanity.com/paper/2510.06219/) (2025)
- [Joint Optimization for 4D Human-Scene Reconstruction in the Wild](https://mlsanity.com/paper/2501.02158/) (2025)
- [HAMSt3R: Human-Aware Multi-view Stereo 3D Reconstruction](https://mlsanity.com/paper/2508.16433/) (2025)
- [UniSH: Unifying Scene and Human Reconstruction in a Feed-Forward Pass](https://mlsanity.com/paper/2601.01222/) (2026)
- [Vision Pretraining for Dense Spatial Perception](https://mlsanity.com/paper/2607.05247/) (2026)
- [E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Models](https://mlsanity.com/paper/2506.01933/) (2025)
- [Humans as a Calibration Pattern: Dynamic 3D Scene Reconstruction from Unsynchronized and Uncalibrated Videos](https://mlsanity.com/paper/2412.19089/) (2024)
- [Kineo: Calibration-Free Metric Motion Capture From Sparse RGB Cameras](https://mlsanity.com/paper/2510.24464/) (2025)
- [OnlineHMR: Video-based Online World-Grounded Human Mesh Recovery](https://mlsanity.com/paper/2603.17355/) (2026)
- [ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine](https://mlsanity.com/paper/2607.28625/) (2026)
- [UAV4D: Dynamic Neural Rendering of Human-Centric UAV Imagery using Gaussian Splatting](https://mlsanity.com/paper/2506.05011/) (2025)
- [A Scalable Whole-body Motion Transfer via Implicit Kinodynamic Motion Retargeting](https://mlsanity.com/paper/2509.15443/) (2025)
- [HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactions](https://mlsanity.com/paper/2603.15612/) (2026)
- [GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction](https://mlsanity.com/paper/2604.19624/) (2026)
- [Visual Sync: Multi-Camera Synchronization via Cross-View Object Motion](https://mlsanity.com/paper/2512.02017/) (2025)
- [EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agents](https://mlsanity.com/paper/2602.23205/) (2026)
- [AHAP: Reconstructing Arbitrary Humans from Arbitrary Perspectives with Geometric Priors](https://mlsanity.com/paper/2602.23951/) (2026)
- [UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video](https://mlsanity.com/paper/2604.19923/) (2026)
- [Unconstrained Multi-view Human Pose Estimation with Algebraic Priors](https://mlsanity.com/paper/2604.24312/) (2026)
- [JOintGS: Joint Optimization of Cameras, Bodies and 3D Gaussians for In-the-Wild Monocular Reconstruction](https://mlsanity.com/paper/2602.04317/) (2026)
- [RPGD: RANSAC-P3P Gradient Descent for Extrinsic Calibration in 3D Human Pose Estimation](https://mlsanity.com/paper/2602.13901/) (2026)
- [HARMONI: Aligning Human and Scene Priors for Multi-View 4D Reconstruction](https://mlsanity.com/paper/2603.12789/) (2026)
- [Multi-Camera Self-Calibration in Sports Motion Capture: Leveraging Human and Stick Poses](https://mlsanity.com/paper/2604.17567/) (2026)
- [RHINO: Reconstructing Human Interactions with Novel Objects from Monocular Videos](https://mlsanity.com/paper/2605.17014/) (2026)
- [TROPHIES: Temporal Reconstruction of Places, Humans, and Cameras from Multi-view Videos](https://mlsanity.com/paper/2606.02350/) (2026)
- [Multi-THuMBS: Multi-person Tracking of 3D Human Meshes Beyond Video Shots](https://mlsanity.com/paper/2607.01626/) (2026)
- [GNOCHI: Generative Neural mOdel for Close Human-Human Interactions](https://mlsanity.com/paper/2607.10408/) (2026)
- [Crowd4D: Scene-Aware Monocular 4D Crowd Reconstruction](https://mlsanity.com/paper/2607.19517/) (2026)
- [Feed-Forward Multi-view Multi-person Reconstruction with Contrastive Human-Aware 3D Representation](https://mlsanity.com/paper/2609.00745/) (2026)

## Abstract (arXiv)

We present "Humans and Structure from Motion" (HSfM), a method for jointly reconstructing multiple human meshes, scene point clouds, and camera parameters in a metric world coordinate system from a sparse set of uncalibrated multi-view images featuring people. Our approach combines data-driven scene reconstruction with the traditional Structure-from-Motion (SfM) framework to achieve more accurate scene reconstruction and camera estimation, while simultaneously recovering human meshes. In contrast to existing scene reconstruction and SfM methods that lack metric scale information, our method estimates approximate metric scale by leveraging a human statistical model. Furthermore, it reconstructs multiple human meshes within the same world coordinate system alongside the scene point cloud, effectively capturing spatial relationships among individuals and their positions in the environment. We initialize the reconstruction of humans, scenes, and cameras using robust foundational models and jointly optimize these elements. This joint optimization synergistically improves the accuracy of each component. We compare our method to existing approaches on two challenging benchmarks, EgoHumans and EgoExo4D, demonstrating significant improvements in human localization accuracy within the world coordinate frame (reducing error from 3.51m to 1.04m in EgoHumans and from 2.9m to 0.56m in EgoExo4D). Notably, our results show that incorporating human data into the SfM pipeline improves camera pose estimation (e.g., increasing RRA@15 by 20.3% on EgoHumans). Additionally, qualitative results show that our approach improves overall scene reconstruction quality. Our code is available at: https://github.com/hongsukchoi/HSfM_RELEASE