mlsanityData to 14 Oct 2026

Grounding Foundational Vision Models with 3D Human Poses for Robust Action Recognition

arXiv 2511.05622 ↗First posted 6 Nov 2025cs.CV cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

For embodied agents to effectively understand and interact within the world around them, they require a nuanced comprehension of human actions grounded in physical space. Current action recognition models, often relying on RGB video, learn superficial correlations between patterns and action labels, so they struggle to capture underlying physical interaction dynamics and human poses in complex scenes. We propose a model architecture that grounds action recognition in physical space by fusing two powerful, complementary representations: V-JEPA 2's contextual, predictive world dynamics and CoMotion's explicit, occlusion-tolerant human pose data. Our model is validated on both the InHARD and UCF-19-Y-OCC benchmarks for general action recognition and high-occlusion action recognition, respectively. Our model outperforms three other baselines, especially within complex, occlusive scenes. Our findings emphasize a need for action recognition to be supported by spatial understanding instead of statistical pattern recognition.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 51 references that are in the map. First posted on arXiv 6 Nov 2025.

Builds on

30 of 51 references.

References · 30

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  4. 2021ViViT: A Video Vision Transformercited by 1,620
  5. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
  6. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  7. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  8. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  9. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  10. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  11. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  12. 2021Revisiting Skeleton-based Action Recognitioncited by 257
18 more
  1. 2020TDN: Temporal Difference Networks for Efficient Action Recognitioncited by 183
  2. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 160
  3. 2020A Survey on 3D Skeleton-Based Action Recognition Using Learning Methodcited by 47
  4. 2024Grounding Multimodal Large Language Models in Actionscited by 35
  5. 2023On the Benefits of 3D Pose and Tracking for Human Action Recognitioncited by 32
  6. 2024Towards Latent Masked Image Modeling for Self-Supervised Visual Representation Learningcited by 20
  7. 2025CoMotion: Concurrent Multi-person 3D Motioncited by 13
  8. 2024Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gatescited by 8
  9. 2024Masked Generative Priors Improve World Models Sequence Modelling Capabilitiescited by 7
  10. 2022Skeleton Sequence and RGB Frame Based Multi-Modality Feature Fusion Network for Action Recognitioncited by 6
  11. 2024Advancing Human Action Recognition with Foundation Models trained on Unlabeled Public Videoscited by 6
  12. 2024Object-Centric Temporal Consistency via Conditional Autoregressive Inductive Biasescited by 5
  13. 2025MV-GMN: State Space Model for Multi-View Action Recognitioncited by 5
  14. 2023A Gated Attention Transformer for Multi-Person Pose Trackingcited by 3
  15. 2025Egocentric Human-Object Interaction Detection: A New Benchmark and Methodcited by 3
  16. 2024ATFusion: An Alternate Cross-Attention Transformer Network for Infrared and Visible Image Fusioncited by 2
  17. 2025SNN-Driven Multimodal Human Action Recognition via Sparse Spatial-Temporal Data Fusioncited by 2
  18. 2025Gated Recursive Fusion: A Stateful Approach to Scalable Multimodal Transformerscited by 2

Built on by

Not cited yet.