# Predicting 3D Human Dynamics from Video

arXiv: https://arxiv.org/abs/1908.04781 · HTML: https://mlsanity.com/paper/1908.04781/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2019.

## Built on by (most cited first, 30 of 51)

- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer](https://mlsanity.com/paper/2303.16160/) (2023)
- [Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders](https://mlsanity.com/paper/2108.06720/) (2021)
- [HumanMAC: Masked Motion Completion for Human Motion Prediction](https://mlsanity.com/paper/2302.03665/) (2023)
- [Physics-based Human Motion Estimation and Synthesis from Videos](https://mlsanity.com/paper/2109.09913/) (2021)
- [Multi-Person 3D Motion Prediction with Multi-Range Transformers](https://mlsanity.com/paper/2111.12073/) (2021)
- [DOVE: Learning Deformable 3D Objects by Watching Videos](https://mlsanity.com/paper/2107.10844/) (2021)
- [Estimating Egocentric 3D Human Pose in Global Space](https://mlsanity.com/paper/2104.13454/) (2021)
- [Object pop-up: Can we infer 3D objects and their poses from human interactions alone?](https://mlsanity.com/paper/2306.00777/) (2023)
- [Understanding 3D Object Articulation in Internet Videos](https://mlsanity.com/paper/2203.16531/) (2022)
- [Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchors](https://mlsanity.com/paper/2302.04860/) (2023)
- [Video Pose Distillation for Few-Shot, Fine-Grained Sports Action Recognition](https://mlsanity.com/paper/2109.01305/) (2021)
- [3D Human Pose, Shape and Texture from Low-Resolution Images and Videos](https://mlsanity.com/paper/2103.06498/) (2021)
- [RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Stream](https://mlsanity.com/paper/2104.14837/) (2021)
- [Weakly-supervised Action Transition Learning for Stochastic Human Motion Prediction](https://mlsanity.com/paper/2205.15608/) (2022)
- [PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos](https://mlsanity.com/paper/2103.09009/) (2021)
- [Tracking People with 3D Representations](https://mlsanity.com/paper/2111.07868/) (2021)
- [Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videos](https://mlsanity.com/paper/2312.13604/) (2023)
- [Tracking People by Predicting 3D Appearance, Location & Pose](https://mlsanity.com/paper/2112.04477/) (2021)
- [H4D: Human 4D Modeling by Learning Neural Compositional Representation](https://mlsanity.com/paper/2203.01247/) (2022)
- [Fine Detailed Texture Learning for 3D Meshes with Generative Models](https://mlsanity.com/paper/2203.09362/) (2022)
- [TrajeVAE: Controllable Human Motion Generation from Trajectories](https://mlsanity.com/paper/2104.00351/) (2021)
- [Audio2Gestures: Generating Diverse Gestures from Audio](https://mlsanity.com/paper/2301.06690/) (2023)
- [Progressive Learning of 3D Reconstruction Network from 2D GAN Data](https://mlsanity.com/paper/2305.11102/) (2023)
- [FIction: 4D Future Interaction Prediction from Video](https://mlsanity.com/paper/2412.00932/) (2024)
- [3D Pose Estimation and Future Motion Prediction from 2D Images](https://mlsanity.com/paper/2111.13285/) (2021)
- [Class-agnostic Reconstruction of Dynamic Objects from Videos](https://mlsanity.com/paper/2112.02091/) (2021)
- [What Happened 3 Seconds Ago? Inferring the Past with Thermal Imaging](https://mlsanity.com/paper/2304.13651/) (2023)
- [Correspondence-free online human motion retargeting](https://mlsanity.com/paper/2302.00556/) (2023)

## Abstract (arXiv)

Given a video of a person in action, we can easily guess the 3D future motion of the person. In this work, we present perhaps the first approach for predicting a future 3D mesh model sequence of a person from past video input. We do this for periodic motions such as walking and also actions like bowling and squatting seen in sports or workout videos. While there has been a surge of future prediction problems in computer vision, most approaches predict 3D future from 3D past or 2D future from 2D past inputs. In this work, we focus on the problem of predicting 3D future motion from past image sequences, which has a plethora of practical applications in autonomous systems that must operate safely around people from visual inputs. Inspired by the success of autoregressive models in language modeling tasks, we learn an intermediate latent space on which we predict the future. This effectively facilitates autoregressive predictions when the input differs from the output domain. Our approach can be trained on video sequences obtained in-the-wild without 3D ground truth labels. The project website with videos can be found at https://jasonyzhang.com/phd.