# VTP: Volumetric Transformer for Multi-view Multi-person 3D Pose Estimation

arXiv: https://arxiv.org/abs/2205.12602 · HTML: https://mlsanity.com/paper/2205.12602/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [Sparse Sinkhorn Attention](https://mlsanity.com/paper/2002.11296/) (2020)
- [Voxel Transformer for 3D Object Detection](https://mlsanity.com/paper/2109.02497/) (2021)
- [Learnable Triangulation of Human Pose](https://mlsanity.com/paper/1905.05754/) (2019)
- [Pose Recognition with Cascade Transformers](https://mlsanity.com/paper/2104.06976/) (2021)
- [Cross View Fusion for 3D Human Pose Estimation](https://mlsanity.com/paper/1909.01203/) (2019)
- [Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views](https://mlsanity.com/paper/1901.04111/) (2019)
- [M3DeTR: Multi-representation, Multi-scale, Mutual-relation 3D Object Detection with Transformers](https://mlsanity.com/paper/2104.11896/) (2021)
- [A Practical Survey on Faster and Lighter Transformers](https://mlsanity.com/paper/2103.14636/) (2021)
- [Direct Multi-view Multi-person 3D Pose Estimation](https://mlsanity.com/paper/2111.04076/) (2021)
- [Parameter Efficient Multimodal Transformers for Video Representation Learning](https://mlsanity.com/paper/2012.04124/) (2020)
- [4D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Cameras](https://mlsanity.com/paper/2002.12625/) (2020)
- [Poseur: Direct Human Pose Regression with Transformers](https://mlsanity.com/paper/2201.07412/) (2022)
- [TFPose: Direct Human Pose Estimation with Transformers](https://mlsanity.com/paper/2103.15320/) (2021)
- [VoxelTrack: Multi-Person 3D Human Pose Estimation and Tracking in the Wild](https://mlsanity.com/paper/2108.02452/) (2021)
- [Graph-Based 3D Multi-Person Pose Estimation Using Multi-View Images](https://mlsanity.com/paper/2109.05885/) (2021)
- [End-to-End Trainable Multi-Instance Pose Estimation with Transformers](https://mlsanity.com/paper/2103.12115/) (2021)
- [Part-Aware Measurement for Robust Multi-View Multi-Human 3D Pose Estimation and Tracking](https://mlsanity.com/paper/2106.11589/) (2021)
- [FreqNet: A Frequency-domain Image Super-Resolution Network with Dicrete Cosine Transform](https://mlsanity.com/paper/2111.10800/) (2021)

## Built on by (most cited first, 3 of 3)

- [Multiple View Geometry Transformers for 3D Human Pose Estimation](https://mlsanity.com/paper/2311.10983/) (2023)
- [Markerless Multi-view 3D Human Pose Estimation: a survey](https://mlsanity.com/paper/2407.03817/) (2024)
- [AvatarPose: Avatar-guided 3D Pose Estimation of Close Human Interaction from Sparse Multi-view Videos](https://mlsanity.com/paper/2408.02110/) (2024)

## Abstract (arXiv)

This paper presents Volumetric Transformer Pose estimator (VTP), the first 3D volumetric transformer framework for multi-view multi-person 3D human pose estimation. VTP aggregates features from 2D keypoints in all camera views and directly learns the spatial relationships in the 3D voxel space in an end-to-end fashion. The aggregated 3D features are passed through 3D convolutions before being flattened into sequential embeddings and fed into a transformer. A residual structure is designed to further improve the performance. In addition, the sparse Sinkhorn attention is empowered to reduce the memory cost, which is a major bottleneck for volumetric representations, while also achieving excellent performance. The output of the transformer is again concatenated with 3D convolutional features by a residual design. The proposed VTP framework integrates the high performance of the transformer with volumetric representations, which can be used as a good alternative to the convolutional backbones. Experiments on the Shelf, Campus and CMU Panoptic benchmarks show promising results in terms of both Mean Per Joint Position Error (MPJPE) and Percentage of Correctly estimated Parts (PCP). Our code will be available.