mlsanityData to 14 Oct 2026

VideoMDM: Towards 3D Human Motion Generation From 2D Supervision

arXiv 2606.13364 ↗First posted 11 Jun 2026cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce VideoMDM, a diffusion-based framework that trains 3D human motion priors directly from accurate 2D poses extracted from monocular videos, without any 3D ground truth. A pretrained 2D-to-3D lifter provides approximate 3D pose sequences that serve as a noisy teacher: these are diffused, denoised by the model in 3D, and supervised in 2D by reprojecting the prediction and comparing against accurate keypoints. We show that, under mild assumptions, a depth-weighted 2D reprojection loss is equivalent in expectation to direct 3D supervision, and we adapt standard 3D motion regularizers - velocity consistency and over-parameterized representation alignment - to this 2D setting. Unlike methods that lift 2D to 3D only at inference, VideoMDM learns a coherent 3D motion manifold during training. On HumanML3D it nearly closes the gap to fully 3D-supervised MDM (FID 0.88 vs 0.54); On real video datasets Fit3D and NBA the method learns to generate motions consistently preferred by humans, with strong quantitative results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 46 of its 55 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

46 of 55 references.

References · 46

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Denoising Diffusion Implicit Modelscited by 9,781
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  9. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  10. 2018Demystifying MMD GANscited by 1,293
  11. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  12. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
34 more
  1. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  2. 2022Human Motion Diffusion Modelcited by 1,080
  3. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  4. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
  5. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  6. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
  7. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 560
  8. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  9. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 521
  10. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 441
  11. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 432
  12. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 400
  13. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  14. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 223
  15. 2023WHAM: Reconstructing World-grounded Humans with Accurate 3D Motioncited by 206
  16. 2022AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Timecited by 200
  17. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 199
  18. 2022MotionBERT: A Unified Perspective on Learning Human Motion Representationscited by 193
  19. 2023RTMPose: Real-Time Multi-Person Pose Estimation based on MMPosecited by 175
  20. 2024TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videoscited by 106
  21. 2023PolyDiff: Generating 3D Polygonal Meshes with Diffusion Modelscited by 66
  22. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  23. 2023Contrastive Denoising Score for Text-guided Latent Diffusion Image Editingcited by 49
  24. 2024RoHM: Robust Human Motion Reconstruction via Diffusioncited by 48
  25. 2024L3DG: Latent 3D Gaussian Diffusioncited by 42
  26. 2024LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesiscited by 39
  27. 2023MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusioncited by 37
  28. 2021ElePose: Unsupervised 3D Human Pose Estimation by Predicting Camera Elevation and Learning Normalizing Flows on 2D Posescited by 35
  29. 2024COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimationcited by 15
  30. 2024Lifting Motion to the 3D World via 2D Diffusioncited by 11
  31. 2024Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filteringcited by 9
  32. 2024A Lesson in Splats: Teacher-Guided Diffusion for 3D Gaussian Splats Generation with 2D Supervisioncited by 9
  33. 2024Motion-2-To-3: Leveraging 2D Motion Data for 3D Motion Generationscited by 9
  34. 2024BiPO: Bidirectional Partial Occlusion Network for Text-to-Motion Synthesiscited by 6

Built on by

Not cited yet.