# TFPose: Direct Human Pose Estimation with Transformers

arXiv: https://arxiv.org/abs/2103.15320 · HTML: https://mlsanity.com/paper/2103.15320/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 53 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation](https://mlsanity.com/paper/1908.10357/) (2019)
- [CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark](https://mlsanity.com/paper/1812.00324/) (2018)
- [Associative Embedding: End-to-End Learning for Joint Detection and Grouping](https://mlsanity.com/paper/1611.05424/) (2016)
- [Towards Accurate Multi-person Pose Estimation in the Wild](https://mlsanity.com/paper/1701.01779/) (2017)
- [Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation](https://mlsanity.com/paper/1406.2984/) (2014)
- [Compositional Human Pose Regression](https://mlsanity.com/paper/1704.00159/) (2017)
- [Human Pose Estimation with Iterative Error Feedback](https://mlsanity.com/paper/1507.06550/) (2015)
- [Single-Stage Multi-Person Pose Machines](https://mlsanity.com/paper/1908.09220/) (2019)
- [The Devil is in the Details: Delving into Unbiased Data Processing for Human Pose Estimation](https://mlsanity.com/paper/1911.07524/) (2019)

## Built on by (most cited first, 30 of 42)

- [DETRs with Hybrid Matching](https://mlsanity.com/paper/2207.13080/) (2022)
- [Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time](https://mlsanity.com/paper/2106.05266/) (2021)
- [Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer](https://mlsanity.com/paper/2204.08680/) (2022)
- [MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction](https://mlsanity.com/paper/2204.07908/) (2022)
- [SimCC: a Simple Coordinate Classification Perspective for Human Pose Estimation](https://mlsanity.com/paper/2107.03332/) (2021)
- [Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2203.04845/) (2022)
- [Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging](https://mlsanity.com/paper/2205.10102/) (2022)
- [Multi-Person 3D Motion Prediction with Multi-Range Transformers](https://mlsanity.com/paper/2111.12073/) (2021)
- [TransCrowd: weakly-supervised crowd counting with transformers](https://mlsanity.com/paper/2104.09116/) (2021)
- [Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction](https://mlsanity.com/paper/2112.13528/) (2021)
- [Explicit Box Detection Unifies End-to-End Multi-Person Pose Estimation](https://mlsanity.com/paper/2302.01593/) (2023)
- [Pose for Everything: Towards Category-Agnostic Pose Estimation](https://mlsanity.com/paper/2207.10387/) (2022)
- [Test-Time Personalization with a Transformer for Human Pose Estimation](https://mlsanity.com/paper/2107.02133/) (2021)
- [TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2110.09554/) (2021)
- [Towards Accurate Facial Landmark Detection via Cascaded Transformers](https://mlsanity.com/paper/2208.10808/) (2022)
- [PPT: token-Pruned Pose Transformer for monocular and multi-view human pose estimation](https://mlsanity.com/paper/2209.08194/) (2022)
- [Vision-Based Human Pose Estimation via Deep Learning: A Survey](https://mlsanity.com/paper/2308.13872/) (2023)
- [Generative Transformer for Accurate and Reliable Salient Object Detection](https://mlsanity.com/paper/2104.10127/) (2021)
- [Temporal Action Proposal Generation with Transformers](https://mlsanity.com/paper/2105.12043/) (2021)
- [A Graph-Based Approach for Category-Agnostic Pose Estimation](https://mlsanity.com/paper/2311.17891/) (2023)
- [Rethinking pose estimation in crowds: overcoming the detection information-bottleneck and ambiguity](https://mlsanity.com/paper/2306.07879/) (2023)
- [DistilPose: Tokenized Pose Regression with Heatmap Distillation](https://mlsanity.com/paper/2303.02455/) (2023)
- [SHaRPose: Sparse High-Resolution Representation for Human Pose Estimation](https://mlsanity.com/paper/2312.10758/) (2023)
- [Boosting Salient Object Detection with Transformer-based Asymmetric Bilateral U-Net](https://mlsanity.com/paper/2108.07851/) (2021)
- [PE-former: Pose Estimation Transformer](https://mlsanity.com/paper/2112.04981/) (2021)
- [Lightweight Super-Resolution Head for Human Pose Estimation](https://mlsanity.com/paper/2307.16765/) (2023)
- [IVT: An End-to-End Instance-guided Video Transformer for 3D Pose Estimation](https://mlsanity.com/paper/2208.03431/) (2022)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)
- [GTPT: Group-based Token Pruning Transformer for Efficient Human Pose Estimation](https://mlsanity.com/paper/2407.10756/) (2024)
- [Referring Human Pose and Mask Estimation in the Wild](https://mlsanity.com/paper/2410.20508/) (2024)

## Abstract (arXiv)

We propose a human pose estimation framework that solves the task in the regression-based fashion. Unlike previous regression-based methods, which often fall behind those state-of-the-art methods, we formulate the pose estimation task into a sequence prediction problem that can effectively be solved by transformers. Our framework is simple and direct, bypassing the drawbacks of the heatmap-based pose estimation. Moreover, with the attention mechanism in transformers, our proposed framework is able to adaptively attend to the features most relevant to the target keypoints, which largely overcomes the feature misalignment issue of previous regression-based methods and considerably improves the performance. Importantly, our framework can inherently take advantages of the structured relationship between keypoints. Experiments on the MS-COCO and MPII datasets demonstrate that our method can significantly improve the state-of-the-art of regression-based pose estimation and perform comparably with the best heatmap-based pose estimation methods.