# Lifting Motion to the 3D World via 2D Diffusion

arXiv: https://arxiv.org/abs/2411.18808 · HTML: https://mlsanity.com/paper/2411.18808/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 67 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model](https://mlsanity.com/paper/2310.15110/) (2023)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior](https://mlsanity.com/paper/2303.14184/) (2023)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2011.14672/) (2020)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [Generative Novel View Synthesis with 3D-Aware Diffusion Models](https://mlsanity.com/paper/2304.02602/) (2023)

## Built on by (most cited first, 11 of 11)

- [Absolute Coordinates Make Motion Generation Easy](https://mlsanity.com/paper/2505.19377/) (2025)
- [Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animation](https://mlsanity.com/paper/2504.19189/) (2025)
- [AnimaX: Animating the Inanimate in 3D with Joint Video-Pose Diffusion Models](https://mlsanity.com/paper/2506.19851/) (2025)
- [Learning to Control Physically-simulated 3D Characters via Generating and Mimicking 2D Motions](https://mlsanity.com/paper/2512.08500/) (2025)
- [AnyLift: Scaling Motion Reconstruction from Internet Videos via 2D Diffusion](https://mlsanity.com/paper/2604.17818/) (2026)
- [PriorFormer: A Transformer for Real-time Monocular 3D Human Pose Estimation with Versatile Geometric Priors](https://mlsanity.com/paper/2508.18238/) (2025)
- [Bimanual 3D Hand Motion and Articulation Forecasting in Everyday Images](https://mlsanity.com/paper/2510.06145/) (2025)
- [LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World](https://mlsanity.com/paper/2605.05390/) (2026)
- [Mocap-2-to-3: Multi-view Lifting for Monocular Motion Recovery with 2D Pretraining](https://mlsanity.com/paper/2503.03222/) (2025)
- [LaxMotion: Rethinking Supervision Granularity for 3D Human Motion Generation](https://mlsanity.com/paper/2511.11368/) (2025)
- [VideoMDM: Towards 3D Human Motion Generation From 2D Supervision](https://mlsanity.com/paper/2606.13364/) (2026)

## Abstract (arXiv)

Estimating 3D motion from 2D observations is a long-standing research challenge. Prior work typically requires training on datasets containing ground truth 3D motions, limiting their applicability to activities well-represented in existing motion capture data. This dependency particularly hinders generalization to out-of-distribution scenarios or subjects where collecting 3D ground truth is challenging, such as complex athletic movements or animal motion. We introduce MVLift, a novel approach to predict global 3D motion -- including both joint rotations and root trajectories in the world coordinate system -- using only 2D pose sequences for training. Our multi-stage framework leverages 2D motion diffusion models to progressively generate consistent 2D pose sequences across multiple views, a key step in recovering accurate global 3D motion. MVLift generalizes across various domains, including human poses, human-object interactions, and animal poses. Despite not requiring 3D supervision, it outperforms prior work on five datasets, including those methods that require 3D supervision.