# Direct Multi-view Multi-person 3D Pose Estimation

arXiv: https://arxiv.org/abs/2111.04076 · HTML: https://mlsanity.com/paper/2111.04076/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 50 references that are in the map.
- First posted on arXiv 7 Nov 2021.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [Exploring Self-attention for Image Recognition](https://mlsanity.com/paper/2004.13621/) (2020)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Panoptic Studio: A Massively Multiview System for Social Interaction Capture](https://mlsanity.com/paper/1612.03153/) (2016)
- [Learnable Triangulation of Human Pose](https://mlsanity.com/paper/1905.05754/) (2019)
- [Coherent Reconstruction of Multiple Humans from a Single Image](https://mlsanity.com/paper/2006.08586/) (2020)
- [PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model](https://mlsanity.com/paper/1803.08225/) (2018)
- [VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera](https://mlsanity.com/paper/1705.01583/) (2017)
- [Lite Transformer with Long-Short Range Attention](https://mlsanity.com/paper/2004.11886/) (2020)
- [PifPaf: Composite Fields for Human Pose Estimation](https://mlsanity.com/paper/1903.06593/) (2019)
- [Epipolar Transformers](https://mlsanity.com/paper/2005.04551/) (2020)
- [Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach](https://mlsanity.com/paper/1704.02447/) (2017)
- [Single-Stage Multi-Person Pose Machines](https://mlsanity.com/paper/1908.09220/) (2019)
- [Cross View Fusion for 3D Human Pose Estimation](https://mlsanity.com/paper/1909.01203/) (2019)
- [PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation](https://mlsanity.com/paper/2105.02465/) (2021)
- [Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views](https://mlsanity.com/paper/1901.04111/) (2019)
- [Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation](https://mlsanity.com/paper/2004.02186/) (2020)

## Built on by (most cited first, 30 of 64)

- [DETRs with Hybrid Matching](https://mlsanity.com/paper/2207.13080/) (2022)
- [Polar Parametrization for Vision-based Surround-View 3D Detection](https://mlsanity.com/paper/2206.10965/) (2022)
- [A Survey on Video Action Recognition in Sports: Datasets, Methods and Applications](https://mlsanity.com/paper/2206.01038/) (2022)
- [PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision](https://mlsanity.com/paper/2203.15625/) (2022)
- [Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDAR](https://mlsanity.com/paper/2211.16951/) (2022)
- [DETR4D: Direct Multi-View 3D Object Detection with Sparse Attention](https://mlsanity.com/paper/2212.07849/) (2022)
- [SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation](https://mlsanity.com/paper/2404.02041/) (2024)
- [Multiple View Geometry Transformers for 3D Human Pose Estimation](https://mlsanity.com/paper/2311.10983/) (2023)
- [SeSDF: Self-evolved Signed Distance Field for Implicit 3D Clothed Human Reconstruction](https://mlsanity.com/paper/2304.00359/) (2023)
- [EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity](https://mlsanity.com/paper/2502.18373/) (2025)
- [Improved Trajectory Reconstruction for Markerless Pose Estimation](https://mlsanity.com/paper/2303.02413/) (2023)
- [Multi-Person 3D Pose and Shape Estimation via Inverse Kinematics and Refinement](https://mlsanity.com/paper/2210.13529/) (2022)
- [Human-M3: A Multi-view Multi-modal Dataset for 3D Human Pose Estimation in Outdoor Scenes](https://mlsanity.com/paper/2308.00628/) (2023)
- [Towards Robust and Smooth 3D Multi-Person Pose Estimation from Monocular Videos in the Wild](https://mlsanity.com/paper/2309.08644/) (2023)
- [Delving Deep into Pixel Alignment Feature for Accurate Multi-view Human Mesh Recovery](https://mlsanity.com/paper/2301.06020/) (2023)
- [POEM: Reconstructing Hand in a Point Embedded Multi-view Stereo](https://mlsanity.com/paper/2304.04038/) (2023)
- [Distribution-Aware Single-Stage Models for Multi-Person 3D Pose Estimation](https://mlsanity.com/paper/2203.07697/) (2022)
- [DV-3DLane: End-to-end Multi-modal 3D Lane Detection with Dual-view Representation](https://mlsanity.com/paper/2406.16072/) (2024)
- [Markerless Multi-view 3D Human Pose Estimation: a survey](https://mlsanity.com/paper/2407.03817/) (2024)
- [MV-SSM: Multi-View State Space Modeling for 3D Human Pose Estimation](https://mlsanity.com/paper/2509.00649/) (2025)
- [FeatAug-DETR: Enriching One-to-Many Matching for DETRs with Feature Augmentation](https://mlsanity.com/paper/2303.01503/) (2023)
- [Probabilistic Triangulation for Uncalibrated Multi-View 3D Human Pose Estimation](https://mlsanity.com/paper/2309.04756/) (2023)
- [AvatarPose: Avatar-guided 3D Pose Estimation of Close Human Interaction from Sparse Multi-view Videos](https://mlsanity.com/paper/2408.02110/) (2024)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)
- [Multi-Person 3D Pose Estimation from Multi-View Uncalibrated Depth Cameras](https://mlsanity.com/paper/2401.15616/) (2024)
- [Multi-view Hand Reconstruction with a Point-Embedded Transformer](https://mlsanity.com/paper/2408.10581/) (2024)
- [Multi-View Pedestrian Occupancy Prediction with a Novel Synthetic Dataset](https://mlsanity.com/paper/2412.13569/) (2024)
- [Graph-based 3D Human Pose Estimation using WiFi Signals](https://mlsanity.com/paper/2511.19105/) (2025)
- [VTP: Volumetric Transformer for Multi-view Multi-person 3D Pose Estimation](https://mlsanity.com/paper/2205.12602/) (2022)
- [Multi-view Human Body Mesh Translator](https://mlsanity.com/paper/2210.01886/) (2022)

## Abstract (arXiv)

We present Multi-view Pose transformer (MvP) for estimating multi-person 3D poses from multi-view images. Instead of estimating 3D joint locations from costly volumetric representation or reconstructing the per-person 3D pose from multiple detected 2D poses as in previous methods, MvP directly regresses the multi-person 3D poses in a clean and efficient way, without relying on intermediate tasks. Specifically, MvP represents skeleton joints as learnable query embeddings and let them progressively attend to and reason over the multi-view information from the input images to directly regress the actual 3D joint locations. To improve the accuracy of such a simple pipeline, MvP presents a hierarchical scheme to concisely represent query embeddings of multi-person skeleton joints and introduces an input-dependent query adaptation approach. Further, MvP designs a novel geometrically guided attention mechanism, called projective attention, to more precisely fuse the cross-view information for each joint. MvP also introduces a RayConv operation to integrate the view-dependent camera geometry into the feature representations for augmenting the projective attention. We show experimentally that our MvP model outperforms the state-of-the-art methods on several benchmarks while being much more efficient. Notably, it achieves 92.3% AP25 on the challenging Panoptic dataset, improving upon the previous best approach [36] by 9.8%. MvP is general and also extendable to recovering human mesh represented by the SMPL model, thus useful for modeling multi-person body shapes. Code and models are available at https://github.com/sail-sg/mvp.