mlsanityData to 7 Oct 2026

IVT: An End-to-End Instance-guided Video Transformer for 3D Pose Estimation

arXiv 2208.03431 ↗First posted 6 Aug 2022cs.CV cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Video 3D human pose estimation aims to localize the 3D coordinates of human joints from videos. Recent transformer-based approaches focus on capturing the spatiotemporal information from sequential 2D poses, which cannot model the contextual depth feature effectively since the visual depth features are lost in the step of 2D pose estimation. In this paper, we simplify the paradigm into an end-to-end framework, Instance-guided Video Transformer (IVT), which enables learning spatiotemporal contextual depth information from visual features effectively and predicts 3D poses directly from video frames. In particular, we firstly formulate video frames as a series of instance-guided tokens and each token is in charge of predicting the 3D pose of a human instance. These tokens contain body structure information since they are extracted by the guidance of joint offsets from the human center to the corresponding body joints. Then, these tokens are sent into IVT for learning spatiotemporal contextual depth. In addition, we propose a cross-scale instance-guided attention mechanism to handle the variational scales among multiple persons. Finally, the 3D poses of each person are decoded from instance-guided tokens by coordinate regression. Experiments on three widely-used 3D pose estimation benchmarks show that the proposed IVT achieves state-of-the-art performances.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 6 Aug 2022.

Builds on

40 of 53 references.

References · 40

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  4. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  5. 2019Objects as Pointscited by 1,172
  6. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  7. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  8. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 421
  9. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  10. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
  11. 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 339
  12. 2018Learning 3D Human Dynamics from Videocited by 315
28 more
  1. 20213D Human Pose Estimation with Spatial and Temporal Transformerscited by 271
  2. 2020Monocular, One-stage, Regression of Multiple 3D Peoplecited by 216
  3. 2016Panoptic Studio: A Massively Multiview System for Social Interaction Capturecited by 173
  4. 2020Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Videocited by 162
  5. 2019Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Imagecited by 131
  6. 2020TransPose: Keypoint Localization via Transformercited by 126
  7. 2019Exploiting temporal context for 3D human pose estimation in the wildcited by 110
  8. 2021Pose Recognition with Cascade Transformerscited by 99
  9. 20203D Human Motion Estimation via Motion Compression and Refinementcited by 98
  10. 2019Single-Stage Multi-Person Pose Machinescited by 95
  11. 2020SRNet: Improving Generalization in 3D Human Pose Estimation with a Split-and-Recombine Approachcited by 94
  12. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  13. 2017Exploiting temporal information for 3D pose estimationcited by 89
  14. 2019Human Mesh Recovery from Monocular Images via a Skeleton-disentangled Representationcited by 87
  15. 20183D Human Pose Estimation in the Wild by Adversarial Learningcited by 86
  16. 2020Motion Guided 3D Pose Estimation from Videoscited by 85
  17. 2019Sim2real transfer learning for 3D human pose estimation: motion to the rescuecited by 75
  18. 20203D Human Pose Estimation using Spatio-Temporal Networks with Explicit Occlusion Trainingcited by 54
  19. 2020SMAP: Single-Shot Multi-Person Absolute 3D Pose Estimationcited by 52
  20. 2020HMOR: Hierarchical Multi-Person Ordinal Relations for Monocular Multi-Person 3D Pose Estimationcited by 44
  21. 2021TFPose: Direct Human Pose Estimation with Transformerscited by 42
  22. 2021Unifying Multimodal Transformer for Bi-directional Image and Text Generationcited by 41
  23. 2020Compressed Volumetric Heatmaps for Multi-Person 3D Pose Estimationcited by 36
  24. 2019Chirality Nets for Human Pose Regressioncited by 31
  25. 2019Trajectory Space Factorization for Deep Video-Based 3D Human Pose Estimationcited by 30
  26. 2021Monocular 3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networkscited by 16
  27. 2022Distribution-Aware Single-Stage Models for Multi-Person 3D Pose Estimationcited by 9
  28. 2020UNOC: Understanding Occlusion for Embodied Presence in Virtual Realitycited by 7

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 2.

6 papers.

  1. 2023PSVT: End-to-End Multi-person 3D Pose and Shape Estimation with Progressive Video Transformerscited by 31
  2. 2023A Single 2D Pose with Context is Worth Hundreds for 3D Human Pose Estimationcited by 11
  3. 2023Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Surveycited by 6
  4. 2024From Histopathology Images to Cell Clouds: Learning Slide Representations with Hierarchical Cell Transformercited by 2
  5. 2023MM-NeRF: Multimodal-Guided 3D Multi-Style Transfer of Neural Radiance Fieldcited by 1
  6. 2024Behavior Imitation for Manipulator Control and Grasping with Deep Reinforcement Learningcited by 0