# Skeletor: Skeletal Transformers for Robust Body-Pose Estimation

arXiv: https://arxiv.org/abs/2104.11712 · HTML: https://mlsanity.com/paper/2104.11712/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 23 Apr 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Convolutional Pose Machines](https://mlsanity.com/paper/1602.00134/) (2016)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Monocular Total Capture: Posing Face, Body, and Hands in the Wild](https://mlsanity.com/paper/1812.01598/) (2018)
- [Progressive Transformers for End-to-End Sign Language Production](https://mlsanity.com/paper/2004.14874/) (2020)
- [Multi-channel Transformers for Multi-articulatory Sign Language Translation](https://mlsanity.com/paper/2009.00299/) (2020)
- [Joey NMT: A Minimalist NMT Toolkit for Novices](https://mlsanity.com/paper/1907.12484/) (2019)
- [Read and Attend: Temporal Localisation in Sign Language Videos](https://mlsanity.com/paper/2103.16481/) (2021)
- [Weakly-Supervised 3D Pose Estimation from a Single Image using Multi-View Consistency](https://mlsanity.com/paper/1909.06119/) (2019)

## Built on by (most cited first, 26 of 26)

- [Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2203.04845/) (2022)
- [SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos](https://mlsanity.com/paper/2112.13715/) (2021)
- [SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2305.04868/) (2023)
- [BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization](https://mlsanity.com/paper/2302.05075/) (2023)
- [BBC-Oxford British Sign Language Dataset](https://mlsanity.com/paper/2111.03635/) (2021)
- [Scaling up Multimodal Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2408.08544/) (2024)
- [PoseBERT: A Generic Transformer Module for Temporal 3D Human Modeling](https://mlsanity.com/paper/2208.10211/) (2022)
- [SignRep: Enhancing Self-Supervised Sign Representations](https://mlsanity.com/paper/2503.08529/) (2025)
- [MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition](https://mlsanity.com/paper/2405.20666/) (2024)
- [Leveraging MoCap Data for Human Mesh Recovery](https://mlsanity.com/paper/2110.09243/) (2021)
- [KPE: Keypoint Pose Encoding for Transformer-based Image Generation](https://mlsanity.com/paper/2203.04907/) (2022)
- [Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2405.05672/) (2024)
- [Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition](https://mlsanity.com/paper/2406.10501/) (2024)
- [Non-Autoregressive Sign Language Production via Knowledge Distillation](https://mlsanity.com/paper/2208.06183/) (2022)
- [Kinematic-aware Hierarchical Attention Network for Human Pose Estimation in Videos](https://mlsanity.com/paper/2211.15868/) (2022)
- [Improving the Robustness of 3D Human Pose Estimation: A Benchmark and Learning from Noisy Input](https://mlsanity.com/paper/2312.06797/) (2023)
- [Denoising Diffusion for 3D Hand Pose Estimation from Images](https://mlsanity.com/paper/2308.09523/) (2023)
- [SkelFormer: Markerless 3D Pose and Shape Estimation using Skeletal Transformers](https://mlsanity.com/paper/2404.12625/) (2024)
- [A Transformer-Based Contrastive Learning Approach for Few-Shot Sign Language Recognition](https://mlsanity.com/paper/2204.02803/) (2022)
- [MoManifold: Learning to Measure 3D Human Motion via Decoupled Joint Acceleration Manifolds](https://mlsanity.com/paper/2409.00736/) (2024)
- [DanceFusion: A Spatio-Temporal Skeleton Diffusion Transformer for Audio-Driven Dance Motion Reconstruction](https://mlsanity.com/paper/2411.04646/) (2024)
- [Biomechanics-Guided Residual Approach to Generalizable Human Motion Generation and Estimation](https://mlsanity.com/paper/2503.06151/) (2025)
- [Cross-Attention Based Influence Model for Manual and Nonmanual Sign Language Analysis](https://mlsanity.com/paper/2409.08162/) (2024)
- [Joint angle based learning to refine kinematic human pose estimation](https://mlsanity.com/paper/2507.11075/) (2025)
- [A Self-Supervised Approach on Motion Calibration for Enhancing Physical Plausibility in Text-to-Motion](https://mlsanity.com/paper/2602.18199/) (2026)
- [Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Production](https://mlsanity.com/paper/2608.09045/) (2026)

## Abstract (arXiv)

Predicting 3D human pose from a single monoscopic video can be highly challenging due to factors such as low resolution, motion blur and occlusion, in addition to the fundamental ambiguity in estimating 3D from 2D. Approaches that directly regress the 3D pose from independent images can be particularly susceptible to these factors and result in jitter, noise and/or inconsistencies in skeletal estimation. Much of which can be overcome if the temporal evolution of the scene and skeleton are taken into account. However, rather than tracking body parts and trying to temporally smooth them, we propose a novel transformer based network that can learn a distribution over both pose and motion in an unsupervised fashion. We call our approach Skeletor. Skeletor overcomes inaccuracies in detection and corrects partial or entire skeleton corruption. Skeletor uses strong priors learn from on 25 million frames to correct skeleton sequences smoothly and consistently. Skeletor can achieve this as it implicitly learns the spatio-temporal context of human motion via a transformer based neural network. Extensive experiments show that Skeletor achieves improved performance on 3D human pose estimation and further provides benefits for downstream tasks such as sign language translation.