mlsanityData to 7 Oct 2026

STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition

arXiv 2303.18177 ↗First posted 31 Mar 2023cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

We study the problem of human action recognition using motion capture (MoCap) sequences. Unlike existing techniques that take multiple manual steps to derive standardized skeleton representations as model input, we propose a novel Spatial-Temporal Mesh Transformer (STMT) to directly model the mesh sequences. The model uses a hierarchical transformer with intra-frame off-set attention and inter-frame self-attention. The attention mechanism allows the model to freely attend between any two vertex patches to learn non-local relationships in the spatial-temporal domain. Masked vertex modeling and future frame prediction are used as two self-supervised tasks to fully activate the bi-directional and auto-regressive attention in our hierarchical transformer. The proposed method achieves state-of-the-art performance compared to skeleton-based and point-cloud-based models on common MoCap benchmarks. Code is available at https://github.com/zgzxy001/STMT.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 42 of its 70 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

42 of 70 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
30 more
  1. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  2. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  3. 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,097
  4. 2020PCT: Point cloud transformercited by 730
  5. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  6. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  7. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  8. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  9. 2018PCN: Point Completion Networkcited by 382
  10. 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 339
  11. 2021Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognitioncited by 317
  12. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  13. 2022NeuMan: Neural Human Radiance Field from a Single Videocited by 244
  14. 2022CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimationcited by 237
  15. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  16. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 179
  17. 2020DiffusionNet: Discretization Agnostic Learning on Surfacescited by 163
  18. 2019MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequencescited by 117
  19. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  20. 2018MeshCNN: A Network with an Edgecited by 69
  21. 2022PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequencescited by 68
  22. 20203DV: 3D Dynamic Voxel for Action Recognition in Depth Videocited by 41
  23. 2016Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networkscited by 32
  24. 2019Skeleton-Based Online Action Prediction Using Scale Selection Networkcited by 20
  25. 2018Depth Pooling Based Large-scale 3D Action Recognition with Convolutional Neural Networkscited by 19
  26. 2018Action Recognition for Depth Video using Multi-view Dynamic Imagescited by 18
  27. 20203DFCNN: Real-Time Action Recognition using 3D Deep Neural Networks with Raw Depth Informationcited by 13
  28. 2021Real-time 3D human action recognition based on Hyperpoint sequencecited by 9
  29. 2020Exploiting the ConvLSTM: Human Action Recognition using Raw Depth Video-Based Recurrent Neural Networkscited by 8
  30. 2022LocATe: End-to-end Localization of Actions in 3D with Transformerscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 2.

7 papers.

  1. 2024Meet JEANIE: a Similarity Measure for 3D Skeleton Sequences via Temporal-Viewpoint Alignmentcited by 10
  2. 2024Storyboard guided Alignment for Fine-grained Video Action Recognitioncited by 3
  3. 2025Jointly Understand Your Command and Intention:Reciprocal Co-Evolution between Scene-Aware 3D Human Motion Synthesis and Analysiscited by 2
  4. 2024G3R: Generating Rich and Fine-grained mmWave Radar Data from 2D Videos for Generalized Gesture Recognitioncited by 1
  5. 2026Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoningcited by 1
  6. 2024Adversarially Masked Video Consistency for Unsupervised Domain Adaptationcited by 0
  7. 2026Assessing Low Back Movement with Motion Tape Sensor Data Through Deep Learningcited by 0