# Gravity-Aware Monocular 3D Human-Object Reconstruction

arXiv: https://arxiv.org/abs/2108.08844 · HTML: https://mlsanity.com/paper/2108.08844/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 52 references that are in the map.
- First posted on arXiv 19 Aug 2021.

## Builds on (most cited first, 30 of 36)

- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [Learning to Estimate 3D Human Pose and Shape from a Single Color Image](https://mlsanity.com/paper/1805.04092/) (2018)
- [Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose](https://mlsanity.com/paper/1611.07828/) (2016)
- [DeepCap: Monocular Human Performance Capture Using Weak Supervision](https://mlsanity.com/paper/2003.08325/) (2020)
- [Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild](https://mlsanity.com/paper/2007.15649/) (2020)
- [Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1907.11346/) (2019)
- [VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera](https://mlsanity.com/paper/1705.01583/) (2017)
- [3D Human Pose Estimation = 2D Pose Estimation + Matching](https://mlsanity.com/paper/1612.06524/) (2016)
- [Unsupervised 3D Pose Estimation with Geometric Self-Supervision](https://mlsanity.com/paper/1904.04812/) (2019)
- [Contact and Human Dynamics from Monocular Video](https://mlsanity.com/paper/2007.11678/) (2020)
- [LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images](https://mlsanity.com/paper/1803.00455/) (2018)
- [RepNet: Weakly Supervised Training of an Adversarial Reprojection Network for 3D Human Pose Estimation](https://mlsanity.com/paper/1902.09868/) (2019)
- [3D Human Pose Estimation from a Single Image via Distance Matrix Regression](https://mlsanity.com/paper/1611.09010/) (2016)
- [Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image](https://mlsanity.com/paper/1701.00295/) (2017)
- [Neural Monocular 3D Human Motion Capture with Physical Awareness](https://mlsanity.com/paper/2105.01057/) (2021)
- [XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera](https://mlsanity.com/paper/1907.00837/) (2019)
- [EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera](https://mlsanity.com/paper/1908.11505/) (2019)
- [C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion](https://mlsanity.com/paper/1909.02533/) (2019)
- [PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time](https://mlsanity.com/paper/2008.08880/) (2020)
- [Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video](https://mlsanity.com/paper/1904.02683/) (2019)
- [In the Wild Human Pose Estimation Using Explicit 2D Features and Intermediate 3D Representations](https://mlsanity.com/paper/1904.03289/) (2019)
- [Discriminative Correlation Filter with Channel and Spatial Reliability](https://mlsanity.com/paper/1611.08461/) (2016)

## Built on by (most cited first, 30 of 30)

- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras](https://mlsanity.com/paper/2112.01524/) (2021)
- [IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions](https://mlsanity.com/paper/2212.07555/) (2022)
- [Human-Aware Object Placement for Visual Environment Reconstruction](https://mlsanity.com/paper/2203.03609/) (2022)
- [HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environment](https://mlsanity.com/paper/2404.00299/) (2024)
- [CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement](https://mlsanity.com/paper/2406.19353/) (2024)
- [D&D: Learning Human Dynamics from Dynamic Camera](https://mlsanity.com/paper/2209.08790/) (2022)
- [Object pop-up: Can we infer 3D objects and their poses from human interactions alone?](https://mlsanity.com/paper/2306.00777/) (2023)
- [PACE: Human and Camera Motion Estimation from in-the-wild Videos](https://mlsanity.com/paper/2310.13768/) (2023)
- [Scene-aware Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2212.11684/) (2022)
- [HULC: 3D Human Motion Capture with Pose Manifold Sampling and Dense Contact Guidance](https://mlsanity.com/paper/2205.05677/) (2022)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [A Survey on Human Interaction Motion Generation](https://mlsanity.com/paper/2503.12763/) (2025)
- [PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarios](https://mlsanity.com/paper/2404.19722/) (2024)
- [ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptors](https://mlsanity.com/paper/2308.12969/) (2023)
- [Scene-Aware 3D Multi-Human Motion Capture from a Single Camera](https://mlsanity.com/paper/2301.05175/) (2023)
- [MoCapDeform: Monocular 3D Human Motion Capture in Deformable Scenes](https://mlsanity.com/paper/2208.08439/) (2022)
- [Decaf: Monocular Deformation Capture for Face and Hand Interactions](https://mlsanity.com/paper/2309.16670/) (2023)
- [Contact-aware Human Motion Generation from Textual Descriptions](https://mlsanity.com/paper/2403.15709/) (2024)
- [Discovering Human-Object Interaction Concepts via Self-Compositional Learning](https://mlsanity.com/paper/2203.14272/) (2022)
- [COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimation](https://mlsanity.com/paper/2408.16426/) (2024)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [Compositional 3D Human-Object Neural Animation](https://mlsanity.com/paper/2304.14070/) (2023)
- [FORCE: Physics-aware Human-object Interaction](https://mlsanity.com/paper/2403.11237/) (2024)
- [3D Human Interaction Generation: A Survey](https://mlsanity.com/paper/2503.13120/) (2025)
- [GraMMaR: Ground-aware Motion Model for 3D Human Motion Reconstruction](https://mlsanity.com/paper/2306.16736/) (2023)
- [Physically Plausible 3D Human-Scene Reconstruction from Monocular RGB Image using an Adversarial Learning Approach](https://mlsanity.com/paper/2307.14570/) (2023)
- [Utilizing Task-Generic Motion Prior to Recover Full-Body Motion from Very Sparse Signals](https://mlsanity.com/paper/2308.15839/) (2023)
- [TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene](https://mlsanity.com/paper/2409.17459/) (2024)
- [Physically Plausible Human-Object Rendering from Sparse Views via 3D Gaussian Splatting](https://mlsanity.com/paper/2503.09640/) (2025)

## Abstract (arXiv)

This paper proposes GraviCap, i.e., a new approach for joint markerless 3D human motion capture and object trajectory estimation from monocular RGB videos. We focus on scenes with objects partially observed during a free flight. In contrast to existing monocular methods, we can recover scale, object trajectories as well as human bone lengths in meters and the ground plane's orientation, thanks to the awareness of the gravity constraining object motions. Our objective function is parametrised by the object's initial velocity and position, gravity direction and focal length, and jointly optimised for one or several free flight episodes. The proposed human-object interaction constraints ensure geometric consistency of the 3D reconstructions and improved physical plausibility of human poses compared to the unconstrained case. We evaluate GraviCap on a new dataset with ground-truth annotations for persons and different objects undergoing free flights. In the experiments, our approach achieves state-of-the-art accuracy in 3D human motion capture on various metrics. We urge the reader to watch our supplementary video. Both the source code and the dataset are released; see http://4dqv.mpi-inf.mpg.de/GraviCap/.