mlsanityData to 7 Oct 2026

LangPose: Language-Aligned Motion for Robust 3D Human Pose Estimation

arXiv 2409.00449 ↗First posted 31 Aug 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

2D-to-3D human pose lifting is an ill-posed problem due to depth ambiguity and occlusion. Existing methods relying on spatial and temporal consistency alone are insufficient to resolve these problems especially in the presence of significant occlusions or high dynamic actions. Semantic information, however, offers a complementary signal that can help disambiguate such cases. To this end, we propose LangPose, a framework that leverages action knowledge by aligning motion embeddings with text embeddings of fine-grained action labels. LangPose operates in two stages: pretraining and fine-tuning. In the pretraining stage, the model simultaneously learns to recognize actions and reconstruct 3D poses from masked and noisy 2D poses. During the fine-tuning stage, the model is further refined using real-world 3D human pose estimation datasets without action labels. Additionally, our framework incorporates masked body parts and masked time windows in motion modeling, encouraging the model to leverage semantic information when spatial and temporal consistency is unreliable. Experiments demonstrate the effectiveness of LangPose, achieving SOTA level performance in 3D pose estimation on public datasets, including Human3.6M and MPI-INF-3DHP. Specifically, LangPose achieves an MPJPE of 36.7mm on Human3.6M with detected 2D poses as input and 15.5mm on MPI-INF-3DHP with ground-truth 2D poses as input.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 59 references that are in the map. First posted on arXiv 31 Aug 2024.

Builds on

39 of 59 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  5. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  6. 2017A Survey on Multi-Task Learningcited by 838
  7. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  8. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  9. 2016Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervisioncited by 493
  10. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  11. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  12. 2017A simple yet effective baseline for 3d human pose estimationcited by 401
27 more
  1. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
  2. 2021Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognitioncited by 317
  3. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  4. 2017Integral Human Pose Regressioncited by 255
  5. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 220
  6. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 194
  7. 2022MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Videocited by 158
  8. 2019Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Imagecited by 131
  9. 20182D/3D Pose Estimation and Action Recognition using Multitask Deep Learningcited by 103
  10. 2022DiffPose: Toward More Reliable 3D Pose Estimationcited by 99
  11. 2020Motion Guided 3D Pose Estimation from Videoscited by 85
  12. 2018View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognitioncited by 84
  13. 2023Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregationcited by 81
  14. 2022P-STMO: Pre-Trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimationcited by 80
  15. 2023ChatPose: Chatting about 3D Human Posecited by 71
  16. 2016Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networkscited by 63
  17. 2023MotionAGFormer: Enhancing 3D Human Pose Estimation with a Transformer-GCNFormer Networkcited by 62
  18. 20203D Human Pose Estimation using Spatio-Temporal Networks with Explicit Occlusion Trainingcited by 54
  19. 2023GLA-GCN: Global-local Adaptive Graph Convolutional Network for 3D Human Pose Estimation from Monocular Videocited by 46
  20. 2024Motion-Agent: A Conversational Framework for Human Motion Generation with LLMscited by 46
  21. 2022Generative Action Description Prompts for Skeleton-based Action Recognitioncited by 43
  22. 2022STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognitioncited by 43
  23. 2019Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognitioncited by 35
  24. 2024KTPFormer: Kinematics and Trajectory Prior Knowledge-Enhanced Transformer for 3D Human Pose Estimationcited by 33
  25. 2021Topology-aware Convolutional Neural Network for Efficient Skeleton-based Action Recognitioncited by 29
  26. 2023Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Languagecited by 11
  27. 2023ActionPrompt: Action-Guided 3D Human Pose Estimation With Text and Pose Promptingcited by 8

Built on by

Not cited yet.