mlsanityData to 14 Oct 2026

Human Part-wise 3D Motion Context Learning for Sign Language Recognition

arXiv 2308.09305 ↗First posted 18 Aug 2023cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose P3D, the human part-wise motion context learning framework for sign language recognition. Our main contributions lie in two dimensions: learning the part-wise motion context and employing the pose ensemble to utilize 2D and 3D pose jointly. First, our empirical observation implies that part-wise context encoding benefits the performance of sign language recognition. While previous methods of sign language recognition learned motion context from the sequence of the entire pose, we argue that such methods cannot exploit part-specific motion context. In order to utilize part-wise motion context, we propose the alternating combination of a part-wise encoding Transformer (PET) and a whole-body encoding Transformer (WET). PET encodes the motion contexts from a part sequence, while WET merges them into a unified context. By learning part-wise motion context, our P3D achieves superior performance on WLASL compared to previous state-of-the-art methods. Second, our framework is the first to ensemble 2D and 3D poses for sign language recognition. Since the 3D pose holds rich motion context and depth information to distinguish the words, our P3D outperformed the previous state-of-the-art methods employing a pose ensemble.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 18 Aug 2023.

Builds on

40 of 53 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  6. 2021ViViT: A Video Vision Transformercited by 1,620
  7. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  8. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  9. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
  10. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
  11. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 418
  12. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 288
28 more
  1. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  2. 20213D Human Pose Estimation with Spatial and Temporal Transformerscited by 272
  3. 2021Revisiting Skeleton-based Action Recognitioncited by 257
  4. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 256
  5. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  6. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 218
  7. 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 179
  8. 2019Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervisioncited by 176
  9. 2022MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Videocited by 159
  10. 2021MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimationcited by 151
  11. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  12. 2016An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Datacited by 126
  13. 2018MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Languagecited by 118
  14. 2018Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregationcited by 106
  15. 2020BSL-1K: Scaling up co-articulated sign language recognition using mouthing cuescited by 106
  16. 2020Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimationcited by 81
  17. 2020Spatial Temporal Transformer Network for Skeleton-based Action Recognitioncited by 77
  18. 2020Transferring Cross-domain Knowledge for Video Sign Language Recognitioncited by 60
  19. 2021SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognitioncited by 60
  20. 2020Fully Convolutional Networks for Continuous Sign Language Recognitioncited by 58
  21. 2021I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modelingcited by 52
  22. 2020Quantitative Survey of the State of the Art in Sign Language Recognitioncited by 50
  23. 2021Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networkscited by 50
  24. 2020Pose-based Sign Language Recognition using GCN and BERTcited by 37
  25. 2019Human Motion Prediction via Learning Local Structure Representations and Temporal Dependenciescited by 34
  26. 2022Skeleton-Parted Graph Scattering Networks for 3D Human Motion Predictioncited by 29
  27. 2022Focal and Global Spatial-Temporal Transformer for Skeleton-based Action Recognitioncited by 18
  28. 2020IntegralAction: Pose-driven Feature Integration for Robust Human Action Recognition in Videoscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 1; 2026: 1.

8 papers.

  1. 2023SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmarkcited by 29
  2. 2024Scaling up Multimodal Pre-training for Sign Language Understandingcited by 21
  3. 2024A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatarscited by 19
  4. 2024MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognitioncited by 13
  5. 2024Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognitioncited by 6
  6. 2024MM-WLAuslan: Multi-View Multi-Modal Word-Level Australian Sign Language Recognition Datasetcited by 6
  7. 2025SegSLR: Promptable Video Segmentation for Isolated Sign Language Recognitioncited by 0
  8. 2026SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognitioncited by 0