# Efficient Vision Transformer for Human Pose Estimation via Patch Selection

arXiv: https://arxiv.org/abs/2306.04225 · HTML: https://mlsanity.com/paper/2306.04225/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 30 of 33)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://mlsanity.com/paper/2003.05997/) (2020)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [Convolutional Pose Machines](https://mlsanity.com/paper/1602.00134/) (2016)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation](https://mlsanity.com/paper/1908.10357/) (2019)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark](https://mlsanity.com/paper/1812.00324/) (2018)
- [Associative Embedding: End-to-End Learning for Joint Detection and Grouping](https://mlsanity.com/paper/1611.05424/) (2016)
- [Adaptive Token Sampling For Efficient Vision Transformers](https://mlsanity.com/paper/2111.15667/) (2021)
- [HRFormer: High-Resolution Transformer for Dense Prediction](https://mlsanity.com/paper/2110.09408/) (2021)
- [TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?](https://mlsanity.com/paper/2106.11297/) (2021)
- [TransPose: Keypoint Localization via Transformer](https://mlsanity.com/paper/2012.14214/) (2020)
- [Pose2Seg: Detection Free Human Instance Segmentation](https://mlsanity.com/paper/1803.10683/) (2018)
- [TokenPose: Learning Keypoint Tokens for Human Pose Estimation](https://mlsanity.com/paper/2104.03516/) (2021)
- [Lite-HRNet: A Lightweight High-Resolution Network](https://mlsanity.com/paper/2104.06403/) (2021)
- [Scalable Vision Transformers with Hierarchical Pooling](https://mlsanity.com/paper/2103.10619/) (2021)
- [AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding](https://mlsanity.com/paper/1711.06475/) (2017)

## Built on by (most cited first, 4 of 4)

- [Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights](https://mlsanity.com/paper/2403.19882/) (2024)
- [When Less is Enough: Adaptive Token Reduction for Efficient Image Representation](https://mlsanity.com/paper/2503.16660/) (2025)
- [Computerized Assessment of Motor Imitation for Distinguishing Autism in Video (CAMI-2DNet)](https://mlsanity.com/paper/2501.08609/) (2025)
- [Transformers with Joint Tokens and Local-Global Attention for Efficient Human Pose Estimation](https://mlsanity.com/paper/2503.00232/) (2025)

## Abstract (arXiv)

While Convolutional Neural Networks (CNNs) have been widely successful in 2D human pose estimation, Vision Transformers (ViTs) have emerged as a promising alternative to CNNs, boosting state-of-the-art performance. However, the quadratic computational complexity of ViTs has limited their applicability for processing high-resolution images. In this paper, we propose three methods for reducing ViT's computational complexity, which are based on selecting and processing a small number of most informative patches while disregarding others. The first two methods leverage a lightweight pose estimation network to guide the patch selection process, while the third method utilizes a set of learnable joint tokens to ensure that the selected patches contain the most important information about body joints. Experiments across six benchmarks show that our proposed methods achieve a significant reduction in computational complexity, ranging from 30% to 44%, with only a minimal drop in accuracy between 0% and 3.5%.