# Self-Supervised 3D Human Pose Estimation in Static Video Via Neural Rendering

arXiv: https://arxiv.org/abs/2210.04514 · HTML: https://mlsanity.com/paper/2210.04514/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 10 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose](https://mlsanity.com/paper/2102.06199/) (2021)
- [Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose](https://mlsanity.com/paper/1611.07828/) (2016)
- [Neural Volumes: Learning Dynamic Renderable Volumes from Images](https://mlsanity.com/paper/1906.07751/) (2019)
- [Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering](https://mlsanity.com/paper/2109.07448/) (2021)
- [SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video](https://mlsanity.com/paper/2201.12792/) (2022)
- [Cross View Fusion for 3D Human Pose Estimation](https://mlsanity.com/paper/1909.01203/) (2019)
- [Exploiting temporal information for 3D pose estimation](https://mlsanity.com/paper/1711.08585/) (2017)
- [Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation](https://mlsanity.com/paper/1804.01110/) (2018)
- [Learning Monocular 3D Human Pose Estimation from Multi-view Images](https://mlsanity.com/paper/1803.04775/) (2018)
- [Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis](https://mlsanity.com/paper/2004.04400/) (2020)
- [Unsupervised Human Pose Estimation through Transforming Shape Templates](https://mlsanity.com/paper/2105.04154/) (2021)
- [Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture](https://mlsanity.com/paper/1808.01525/) (2018)
- [Recurrent 3D Pose Sequence Machines](https://mlsanity.com/paper/1707.09695/) (2017)
- [A Dual-Source Approach for 3D Human Pose Estimation from a Single Image](https://mlsanity.com/paper/1705.02883/) (2017)
- [Learning Foreground-Background Segmentation from Improved Layered GANs](https://mlsanity.com/paper/2104.00483/) (2021)

## Built on by (most cited first, 2 of 2)

- [MEGA: Masked Generative Autoencoder for Human Mesh Recovery](https://mlsanity.com/paper/2405.18839/) (2024)
- [SSPINNpose: A Self-Supervised PINN for Inertial Pose and Dynamics Estimation](https://mlsanity.com/paper/2506.11786/) (2025)

## Abstract (arXiv)

Inferring 3D human pose from 2D images is a challenging and long-standing problem in the field of computer vision with many applications including motion capture, virtual reality, surveillance or gait analysis for sports and medicine. We present preliminary results for a method to estimate 3D pose from 2D video containing a single person and a static background without the need for any manual landmark annotations. We achieve this by formulating a simple yet effective self-supervision task: our model is required to reconstruct a random frame of a video given a frame from another timepoint and a rendered image of a transformed human shape template. Crucially for optimisation, our ray casting based rendering pipeline is fully differentiable, enabling end to end training solely based on the reconstruction task.