# Uplift and Upsample: Efficient 3D Human Pose Estimation with Uplifting Transformers

arXiv: https://arxiv.org/abs/2210.06110 · HTML: https://mlsanity.com/paper/2210.06110/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 60 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [3D Human Pose Estimation with Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.10455/) (2021)
- [HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation](https://mlsanity.com/paper/1908.10357/) (2019)
- [MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video](https://mlsanity.com/paper/2203.00859/) (2022)
- [MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2111.12707/) (2021)
- [VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera](https://mlsanity.com/paper/1705.01583/) (2017)
- [TokenPose: Learning Keypoint Tokens for Human Pose Estimation](https://mlsanity.com/paper/2104.03516/) (2021)
- [2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning](https://mlsanity.com/paper/1802.09232/) (2018)
- [Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2103.14304/) (2021)
- [PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation](https://mlsanity.com/paper/2105.02465/) (2021)
- [RepNet: Weakly Supervised Training of an Adversarial Reprojection Network for 3D Human Pose Estimation](https://mlsanity.com/paper/1902.09868/) (2019)

## Built on by (most cited first, 30 of 33)

- [PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation](https://mlsanity.com/paper/2303.17472/) (2023)
- [MotionAGFormer: Enhancing 3D Human Pose Estimation with a Transformer-GCNFormer Network](https://mlsanity.com/paper/2310.16288/) (2023)
- [3D Human Pose Perception from Egocentric Stereo Videos](https://mlsanity.com/paper/2401.00889/) (2023)
- [Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation](https://mlsanity.com/paper/2311.12028/) (2023)
- [AthletePose3D: A Benchmark Dataset for 3D Human Pose Estimation and Kinematic Validation in Athletic Movements](https://mlsanity.com/paper/2503.07499/) (2025)
- [Pose Magic: Efficient and Temporally Consistent Human Pose Estimation with a Hybrid Mamba-GCN Network](https://mlsanity.com/paper/2408.02922/) (2024)
- [Regular Splitting Graph Network for 3D Human Pose Estimation](https://mlsanity.com/paper/2305.05785/) (2023)
- [A Single 2D Pose with Context is Worth Hundreds for 3D Human Pose Estimation](https://mlsanity.com/paper/2311.03312/) (2023)
- [Bring Your Rear Cameras for Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2503.11652/) (2025)
- [HumMUSS: Human Motion Understanding using State Space Models](https://mlsanity.com/paper/2404.10880/) (2024)
- [H2OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers](https://mlsanity.com/paper/2509.06956/) (2025)
- [EgoPoseFormer: A Simple Baseline for Stereo Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2403.18080/) (2024)
- [HGMamba: Enhancing 3D Human Pose Estimation with a HyperGCN-Mamba Network](https://mlsanity.com/paper/2504.06638/) (2025)
- [Towards Ball Spin and Trajectory Analysis in Table Tennis Broadcast Videos via Physically Grounded Synthetic-to-Real Transfer](https://mlsanity.com/paper/2504.19863/) (2025)
- [STGFormer: Spatio-Temporal GraphFormer for 3D Human Pose Estimation in Video](https://mlsanity.com/paper/2407.10099/) (2024)
- [Enhancing 3D Human Pose Estimation Amidst Severe Occlusion with Dual Transformer Fusion](https://mlsanity.com/paper/2410.04574/) (2024)
- [A Structure-aware and Motion-adaptive Framework for 3D Human Pose Estimation with Mamba](https://mlsanity.com/paper/2507.19852/) (2025)
- [Leveraging Anthropometric Measurements to Improve Human Mesh Estimation and Ensure Consistent Body Shapes](https://mlsanity.com/paper/2409.17671/) (2024)
- [SCJD: Sparse Correlation and Joint Distillation for Efficient 3D Human Pose Estimation](https://mlsanity.com/paper/2503.14097/) (2025)
- [Mamba-Driven Topology Fusion for Monocular 3D Human Pose Estimation](https://mlsanity.com/paper/2505.20611/) (2025)
- [Efficient, Self-Supervised Human Pose Estimation with Inductive Prior Tuning](https://mlsanity.com/paper/2311.02815/) (2023)
- [Graph and Skipped Transformer: Exploiting Spatial and Temporal Modeling Capacities for Efficient 3D Human Pose Estimation](https://mlsanity.com/paper/2407.02990/) (2024)
- [Efficient 2D to Full 3D Human Pose Uplifting including Joint Rotations](https://mlsanity.com/paper/2504.09953/) (2025)
- [Spectral Compression Transformer with Line Pose Graph for Monocular 3D Human Pose Estimation](https://mlsanity.com/paper/2505.21309/) (2025)
- [PriorFormer: A Transformer for Real-time Monocular 3D Human Pose Estimation with Versatile Geometric Priors](https://mlsanity.com/paper/2508.18238/) (2025)
- [SasMamba: A Lightweight Structure-Aware Stride State Space Model for 3D Human Pose Estimation](https://mlsanity.com/paper/2511.08872/) (2025)
- [TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos](https://mlsanity.com/paper/2605.01234/) (2026)
- [Learning Human Motion from Monocular Videos via Cross-Modal Manifold Alignment](https://mlsanity.com/paper/2404.09499/) (2024)
- [VioPose: Violin Performance 4D Pose Estimation by Hierarchical Audiovisual Inference](https://mlsanity.com/paper/2411.13607/) (2024)
- [HDiffTG: A Lightweight Hybrid Diffusion-Transformer-GCN Architecture for 3D Human Pose Estimation](https://mlsanity.com/paper/2505.04276/) (2025)

## Abstract (arXiv)

The state-of-the-art for monocular 3D human pose estimation in videos is dominated by the paradigm of 2D-to-3D pose uplifting. While the uplifting methods themselves are rather efficient, the true computational complexity depends on the per-frame 2D pose estimation. In this paper, we present a Transformer-based pose uplifting scheme that can operate on temporally sparse 2D pose sequences but still produce temporally dense 3D pose estimates. We show how masked token modeling can be utilized for temporal upsampling within Transformer blocks. This allows to decouple the sampling rate of input 2D poses and the target frame rate of the video and drastically decreases the total computational complexity. Additionally, we explore the option of pre-training on large motion capture archives, which has been largely neglected so far. We evaluate our method on two popular benchmark datasets: Human3.6M and MPI-INF-3DHP. With an MPJPE of 45.0 mm and 46.9 mm, respectively, our proposed method can compete with the state-of-the-art while reducing inference time by a factor of 12. This enables real-time throughput with variable consumer hardware in stationary and mobile applications. We release our code and models at https://github.com/goldbricklemon/uplift-upsample-3dhpe