mlsanityData to 7 Oct 2026

Learning Human Motion from Monocular Videos via Cross-Modal Manifold Alignment

arXiv 2404.09499 ↗First posted 15 Apr 2024cs.CV cs.GRcited by 0

Not on the map. Why?

Abstract (arXiv)

Learning 3D human motion from 2D inputs is a fundamental task in the realms of computer vision and computer graphics. Many previous methods grapple with this inherently ambiguous task by introducing motion priors into the learning process. However, these approaches face difficulties in defining the complete configurations of such priors or training a robust model. In this paper, we present the Video-to-Motion Generator (VTM), which leverages motion priors through cross-modal latent feature space alignment between 3D human motion and 2D inputs, namely videos and 2D keypoints. To reduce the complexity of modeling motion priors, we model the motion data separately for the upper and lower body parts. Additionally, we align the motion data with a scale-invariant virtual skeleton to mitigate the interference of human skeleton variations to the motion priors. Evaluated on AIST++, the VTM showcases state-of-the-art performance in reconstructing 3D human motion from monocular videos. Notably, our VTM exhibits the capabilities for generalization to unseen view angles and in-the-wild videos.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 63 of its 86 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

63 of 86 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  6. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  7. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  8. 2017End-to-end Recovery of Human Shape and Posecited by 883
  9. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  10. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  11. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  12. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 573
38 more
  1. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  2. 2017A simple yet effective baseline for 3d human pose estimationcited by 401
  3. 20213D Human Pose Estimation with Spatial and Temporal Transformerscited by 271
  4. 2017Integral Human Pose Regressioncited by 255
  5. 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
  6. 2016Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Posecited by 220
  7. 2020Monocular, One-stage, Regression of Multiple 3D Peoplecited by 216
  8. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 210
  9. 2017Unite the People: Closing the Loop Between 3D and 2D Human Representationscited by 205
  10. 2020Deep Learning-Based Human Pose Estimation: A Surveycited by 205
  11. 2019Semantic Graph Convolutional Networks for 3D Human Pose Regressioncited by 204
  12. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  13. 2021Character Controllers Using Motion VAEscited by 182
  14. 2022MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Videocited by 158
  15. 2021MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimationcited by 151
  16. 2021Putting People in their Place: Monocular Regression of 3D People in Depthcited by 141
  17. 2021Human Pose Regression with Residual Log-likelihood Estimationcited by 131
  18. 2017Compositional Human Pose Regressioncited by 124
  19. 20182D/3D Pose Estimation and Action Recognition using Multitask Deep Learningcited by 103
  20. 2019Unsupervised 3D Pose Estimation with Geometric Self-Supervisioncited by 95
  21. 2019Single-Stage Multi-Person Pose Machinescited by 95
  22. 2021Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimationcited by 94
  23. 2023PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimationcited by 91
  24. 2021PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimationcited by 88
  25. 20183D Human Pose Estimation in the Wild by Adversarial Learningcited by 86
  26. 2019RepNet: Weakly Supervised Training of an Adversarial Reprojection Network for 3D Human Pose Estimationcited by 85
  27. 2022P-STMO: Pre-Trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimationcited by 80
  28. 20163D Human Pose Estimation from a Single Image via Distance Matrix Regressioncited by 73
  29. 2017Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Imagecited by 73
  30. 2018Numerical Coordinate Regression with Convolutional Neural Networkscited by 72
  31. 2020Monocular Human Pose Estimation: A Survey of Deep Learning-based Methodscited by 59
  32. 2022GFPose: Learning 3D Human Pose Prior with Gradient Fieldscited by 57
  33. 2017Human Pose Regression by Combining Indirect Part Detection and Contextual Informationcited by 52
  34. 2019HEMlets Pose: Learning Part-Centric Heatmap Triplets for Accurate 3D Human Pose Estimationcited by 50
  35. 2019In the Wild Human Pose Estimation Using Explicit 2D Features and Intermediate 3D Representationscited by 48
  36. 2021Body Meshes as Pointscited by 45
  37. 2023Human Pose as Compositional Tokenscited by 45
  38. 2016Deep Kinematic Pose Regressioncited by 40

50 most cited of 63 All 63 (JSON)

Built on by

Not cited yet.