# PACE: Human and Camera Motion Estimation from in-the-wild Videos

arXiv: https://arxiv.org/abs/2310.13768 · HTML: https://mlsanity.com/paper/2310.13768/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 107 of its 130 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 30 of 107)

- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2011.14672/) (2020)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop](https://mlsanity.com/paper/2103.16507/) (2021)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [Diffusion-based Generation, Optimization, and Planning in 3D Scenes](https://mlsanity.com/paper/2301.06015/) (2023)
- [CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation](https://mlsanity.com/paper/2208.00571/) (2022)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)

## Built on by (most cited first, 30 of 34)

- [TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videos](https://mlsanity.com/paper/2403.17346/) (2024)
- [Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generation](https://mlsanity.com/paper/2504.14899/) (2025)
- [PromptHMR: Promptable Human Mesh Recovery](https://mlsanity.com/paper/2504.06397/) (2025)
- [Visual Imitation Enables Contextual Humanoid Control](https://mlsanity.com/paper/2505.03729/) (2025)
- [HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos](https://mlsanity.com/paper/2501.02973/) (2025)
- [Human3R: Everyone Everywhere All at Once](https://mlsanity.com/paper/2510.06219/) (2025)
- [WHAC: World-grounded Humans and Cameras](https://mlsanity.com/paper/2403.12959/) (2024)
- [Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mind](https://mlsanity.com/paper/2404.05072/) (2024)
- [BEDLAM2.0: Synthetic Humans and Cameras in Motion](https://mlsanity.com/paper/2511.14394/) (2025)
- [CoMotion: Concurrent Multi-person 3D Motion](https://mlsanity.com/paper/2504.12186/) (2025)
- [MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos](https://mlsanity.com/paper/2512.10881/) (2025)
- [UniSH: Unifying Scene and Human Reconstruction in a Feed-Forward Pass](https://mlsanity.com/paper/2601.01222/) (2026)
- [Pulp Motion: Framing-aware multimodal camera and human motion generation](https://mlsanity.com/paper/2510.05097/) (2025)
- [Humans as a Calibration Pattern: Dynamic 3D Scene Reconstruction from Unsynchronized and Uncalibrated Videos](https://mlsanity.com/paper/2412.19089/) (2024)
- [Humans as Checkerboards: Calibrating Camera Motion Scale for World-Coordinate Human Mesh Recovery](https://mlsanity.com/paper/2407.00574/) (2024)
- [Towards Motion Turing Test: Evaluating Human-Likeness in Humanoid Robots](https://mlsanity.com/paper/2603.06181/) (2026)
- [Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos](https://mlsanity.com/paper/2410.16259/) (2024)
- [ClimbingCap: Multi-Modal Dataset and Method for Rock Climbing in World Coordinate](https://mlsanity.com/paper/2503.21268/) (2025)
- [DyCrowd: Towards Dynamic Crowd Reconstruction from a Large-scene Video](https://mlsanity.com/paper/2508.12644/) (2025)
- [Stereo-Inertial Poser: Towards Metric-Accurate Shape-Aware Motion Capture Using Sparse IMUs and a Single Stereo Camera](https://mlsanity.com/paper/2603.02130/) (2026)
- [SSPINNpose: A Self-Supervised PINN for Inertial Pose and Dynamics Estimation](https://mlsanity.com/paper/2506.11786/) (2025)
- [UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video](https://mlsanity.com/paper/2604.19923/) (2026)
- [LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World](https://mlsanity.com/paper/2605.05390/) (2026)
- [EgoExoMoCap: Distributed Ego-Exo Human Motion Capture](https://mlsanity.com/paper/2607.15868/) (2026)
- [MAMMA: Markerless & Automatic Multi-Person Motion Action Capture](https://mlsanity.com/paper/2506.13040/) (2025)
- [VirtualFencer: Generating Fencing Bouts based on Strategies Extracted from In-the-Wild Videos](https://mlsanity.com/paper/2507.00261/) (2025)
- [Physics-based Human Pose Estimation from a Single Moving RGB Camera](https://mlsanity.com/paper/2507.17406/) (2025)
- [WATCH: World-aware Allied Trajectory and pose reConstruction for Camera and Human](https://mlsanity.com/paper/2509.04600/) (2025)
- [Improving Human Motion Plausibility with Body Momentum](https://mlsanity.com/paper/2509.09496/) (2025)
- [Improving 3D Foot Motion Reconstruction in Markerless Monocular Human Motion Capture](https://mlsanity.com/paper/2603.09681/) (2026)

## Abstract (arXiv)

We present a method to estimate human motion in a global scene from moving cameras. This is a highly challenging task due to the coupling of human and camera motions in the video. To address this problem, we propose a joint optimization framework that disentangles human and camera motions using both foreground human motion priors and background scene features. Unlike existing methods that use SLAM as initialization, we propose to tightly integrate SLAM and human motion priors in an optimization that is inspired by bundle adjustment. Specifically, we optimize human and camera motions to match both the observed human pose and scene features. This design combines the strengths of SLAM and motion priors, which leads to significant improvements in human and camera motion estimation. We additionally introduce a motion prior that is suitable for batch optimization, making our approach significantly more efficient than existing approaches. Finally, we propose a novel synthetic dataset that enables evaluating camera motion in addition to human motion from dynamic videos. Experiments on the synthetic and real-world RICH datasets demonstrate that our approach substantially outperforms prior art in recovering both human and camera motions.