# Building Spatio-temporal Transformers for Egocentric 3D Pose Estimation

arXiv: https://arxiv.org/abs/2206.04785 · HTML: https://mlsanity.com/paper/2206.04785/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 16 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [3D Human Pose Estimation with Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.10455/) (2021)
- [Compositional Human Pose Regression](https://mlsanity.com/paper/1704.00159/) (2017)
- [3D Human Pose Estimation = 2D Pose Estimation + Matching](https://mlsanity.com/paper/1612.06524/) (2016)
- [Ordinal Depth Supervision for 3D Human Pose Estimation](https://mlsanity.com/paper/1805.04095/) (2018)
- [xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera](https://mlsanity.com/paper/1907.10045/) (2019)
- [Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera](https://mlsanity.com/paper/1803.05959/) (2018)
- [Estimating Egocentric 3D Human Pose in Global Space](https://mlsanity.com/paper/2104.13454/) (2021)
- [HEMlets Pose: Learning Part-Centric Heatmap Triplets for Accurate 3D Human Pose Estimation](https://mlsanity.com/paper/1910.12032/) (2019)
- [Structured Prediction of 3D Human Pose with Deep Neural Networks](https://mlsanity.com/paper/1605.05180/) (2016)
- [Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation](https://mlsanity.com/paper/1508.06708/) (2015)
- [Scoring and Assessment in Medical VR Training Simulators with Dynamic Time Series Classification](https://mlsanity.com/paper/2006.12366/) (2020)
- [Architectural Visualization Using Virtual Reality: A User Experience in Simulating Buildings of a Community College in Bukidnon, Philippines](https://mlsanity.com/paper/2103.06238/) (2021)

## Built on by (most cited first, 2 of 2)

- [EgoPoseFormer: A Simple Baseline for Stereo Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2403.18080/) (2024)
- [EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR](https://mlsanity.com/paper/2603.04090/) (2026)

## Abstract (arXiv)

Egocentric 3D human pose estimation (HPE) from images is challenging due to severe self-occlusions and strong distortion introduced by the fish-eye view from the head mounted camera. Although existing works use intermediate heatmap-based representations to counter distortion with some success, addressing self-occlusion remains an open problem. In this work, we leverage information from past frames to guide our self-attention-based 3D HPE estimation procedure -- Ego-STAN. Specifically, we build a spatio-temporal Transformer model that attends to semantically rich convolutional neural network-based feature maps. We also propose feature map tokens: a new set of learnable parameters to attend to these feature maps. Finally, we demonstrate Ego-STAN's superior performance on the xR-EgoPose dataset where it achieves a 30.6% improvement on the overall mean per-joint position error, while leading to a 22% drop in parameters compared to the state-of-the-art.