mlsanityData to 7 Oct 2026

On the Benefits of 3D Pose and Tracking for Human Action Recognition

arXiv 2304.01199 ↗First posted 3 Apr 2023cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

In this work we study the benefits of using tracking and 3D poses for action recognition. To achieve this, we take the Lagrangian view on analysing actions over a trajectory of human motion rather than at a fixed point in space. Taking this stand allows us to use the tracklets of people to predict their actions. In this spirit, first we show the benefits of using 3D pose to infer actions, and study person-person interactions. Subsequently, we propose a Lagrangian Action Recognition model by fusing 3D pose and contextualized appearance over tracklets. To this end, our method achieves state-of-the-art performance on the AVA v2.2 dataset on both pose only settings and on standard benchmark settings. When reasoning about the action using only pose cues, our pose model achieves +10.0 mAP gain over the corresponding state-of-the-art while our fused model has a gain of +2.8 mAP over the best state-of-the-art model. Code and results are available at: https://brjathu.github.io/LART

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 54 of its 74 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

54 of 74 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2017Mask R-CNNcited by 7,259
  5. 2016Layer Normalizationcited by 5,579
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2017Non-local Neural Networkscited by 2,290
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2018SlowFast Networks for Video Recognitioncited by 1,862
  11. 2021ViViT: A Video Vision Transformercited by 1,610
  12. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
38 more
  1. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  2. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  3. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  4. 2016Deep Networks with Stochastic Depthcited by 973
  5. 2020Designing Network Design Spacescited by 954
  6. 2021Multiscale Vision Transformerscited by 896
  7. 2017End-to-end Recovery of Human Shape and Posecited by 883
  8. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  9. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  10. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  11. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  12. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  13. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  14. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  15. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  16. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  17. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  18. 2019Tracking without bells and whistlescited by 382
  19. 2016RMPE: Regional Multi-person Pose Estimationcited by 340
  20. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  21. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  22. 2018Learning 3D Human Dynamics from Videocited by 315
  23. 2021Video Transformer Networkcited by 271
  24. 2021PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loopcited by 250
  25. 2018Videos as Space-Time Region Graphscited by 221
  26. 2021Probabilistic Modeling for Human Mesh Recoverycited by 152
  27. 2021Towards Long-Form Video Understandingcited by 146
  28. 2021SPEC: Seeing People in the Wild with an Estimated Cameracited by 134
  29. 2020Actor-Context-Actor Relation Network for Spatio-Temporal Action Localizationcited by 91
  30. 2020The AVA-Kinetics Localized Human Actions Video Datasetcited by 82
  31. 2018Pose Flow: Efficient Online Pose Trackingcited by 80
  32. 2020Human Mesh Recovery from Multiple Shotscited by 52
  33. 2014Finding Action Tubescited by 50
  34. 2022Accurate 3D Body Shape Regression using Metric and Semantic Attributescited by 49
  35. 2019Synthetic Humans for Action Recognition from Unseen Viewpointscited by 46
  36. 2018A Structured Model For Action Detectioncited by 43
  37. 2019Mimetics: Towards Understanding Human Actions Out of Contextcited by 32
  38. 2019Relational Action Forecastingcited by 27

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 12; 2025: 12; 2026: 5.

32 papers.

Sort:

By area:

  1. 2024Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shotcited by 98
  2. 2025Visual Imitation Enables Contextual Humanoid Controlcited by 57
  3. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Modelcited by 21
  4. 2024On the Utility of 3D Hand Poses for Action Recognitioncited by 17
  5. 2024Referring Atomic Video Action Recognitioncited by 16
  6. 2023VideoGLUE: Video General Understanding Evaluation of Foundation Modelscited by 14
  7. 2023SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolationcited by 11
  8. 2024Computer Vision for Primate Behavior Analysis in the Wildcited by 10
  9. 2024Recurrent Aligned Network for Generalized Pedestrian Trajectory Predictioncited by 8
  10. 2023Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Surveycited by 6
  11. 2024Enhancing Video Transformers for Action Understanding with VLM-aided Trainingcited by 6
  12. 2024Skeleton-based Group Activity Recognition via Spatial-Temporal Panoramic Graphcited by 6
20 more
  1. 2025HuMoCon: Concept Discovery for Human Motion Understandingcited by 4
  2. 2024Enhancing 2D Representation Learning with a 3D Priorcited by 3
  3. 2024CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognitioncited by 3
  4. 2025Egocentric Human-Object Interaction Detection: A New Benchmark and Methodcited by 3
  5. 2024BID: Boundary-Interior Decoding for Unsupervised Temporal Action Localization Pre-Trainincited by 2
  6. 2024ChildPlay-Hand: A Dataset of Hand Manipulations in the Wildcited by 2
  7. 2025SCJD: Sparse Correlation and Joint Distillation for Efficient 3D Human Pose Estimationcited by 2
  8. 2025MMHOI: Modeling Complex 3D Multi-Human Multi-Object Interactionscited by 2
  9. 2025StarPose: 3D Human Pose Estimation via Spatial-Temporal Autoregressive Diffusioncited by 1
  10. 2025Age-Inclusive 3D Human Mesh Recovery for Action-Preserving Data Anonymizationcited by 1
  11. 2026BigMaQ: A Big Macaque Motion and Animation Dataset Bridging Image and 3D Pose Representationscited by 1
  12. 2025ViewBridge: Curriculum Knowledge Distillation for Activity View-Invariance Under Extreme Viewpoint Changescited by 0
  13. 2025HDiffTG: A Lightweight Hybrid Diffusion-Transformer-GCN Architecture for 3D Human Pose Estimationcited by 0
  14. 2025RefAtomNet++: Advancing Referring Atomic Video Action Recognition using Semantic Retrieval based Multi-Trajectory Mambacited by 0
  15. 2025Grounding Foundational Vision Models with 3D Human Poses for Robust Action Recognitioncited by 0
  16. 2025Improving action classification with brain-inspired deep networkscited by 0
  17. 2026PoseDriver: A Unified Approach to Multi-Category Skeleton Detection for Autonomous Drivingcited by 0
  18. 2026TaskNPoint: How to Teach Your Humanoid to Hit a Backhand in Minutescited by 0
  19. 2026STAR: Skeletal Token Alignment and Rearrangement for Interaction Recognitioncited by 0
  20. 2026De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shiftcited by 0