# PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision

arXiv: https://arxiv.org/abs/2203.15625 · HTML: https://mlsanity.com/paper/2203.15625/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 67 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 54)

- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [Robust Motion In-betweening](https://mlsanity.com/paper/2102.04942/) (2021)
- [DLow: Diversifying Latent Flows for Diverse Human Motion Prediction](https://mlsanity.com/paper/2003.08386/) (2020)
- [HP-GAN: Probabilistic 3D human motion prediction via GAN](https://mlsanity.com/paper/1711.09561/) (2017)
- [An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1902.09130/) (2019)
- [Coherent Reconstruction of Multiple Humans from a Single Image](https://mlsanity.com/paper/2006.08586/) (2020)
- [Distilling Knowledge from Graph Convolutional Networks](https://mlsanity.com/paper/2003.10477/) (2020)
- [Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach](https://mlsanity.com/paper/1704.02447/) (2017)
- [Convolutional Sequence to Sequence Model for Human Dynamics](https://mlsanity.com/paper/1805.00655/) (2018)
- [We are More than Our Joints: Predicting how 3D Bodies Move](https://mlsanity.com/paper/2012.00619/) (2020)
- [Human Motion Prediction via Spatio-Temporal Inpainting](https://mlsanity.com/paper/1812.05478/) (2018)
- [SimPoE: Simulated Character Control for 3D Human Pose Estimation](https://mlsanity.com/paper/2104.00683/) (2021)
- [QuaterNet: A Quaternion-based Recurrent Model for Human Motion](https://mlsanity.com/paper/1805.06485/) (2018)
- [Deep representation learning for human motion prediction and classification](https://mlsanity.com/paper/1702.07486/) (2017)
- [Self-Supervised Learning of 3D Human Pose using Multi-view Geometry](https://mlsanity.com/paper/1903.02330/) (2019)
- [Unsupervised 3D Pose Estimation with Geometric Self-Supervision](https://mlsanity.com/paper/1904.04812/) (2019)
- [Single-Stage Multi-Person Pose Machines](https://mlsanity.com/paper/1908.09220/) (2019)
- [Contact and Human Dynamics from Monocular Video](https://mlsanity.com/paper/2007.11678/) (2020)
- [PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation](https://mlsanity.com/paper/2105.02465/) (2021)
- [Human Mesh Recovery from Monocular Images via a Skeleton-disentangled Representation](https://mlsanity.com/paper/1908.07172/) (2019)
- [Cascaded deep monocular 3D human pose estimation with evolutionary training data](https://mlsanity.com/paper/2006.07778/) (2020)
- [Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation](https://mlsanity.com/paper/1804.01110/) (2018)
- [Convolutional Autoencoders for Human Motion Infilling](https://mlsanity.com/paper/2010.11531/) (2020)
- [Neural Monocular 3D Human Motion Capture with Physical Awareness](https://mlsanity.com/paper/2105.01057/) (2021)

## Built on by (most cited first, 25 of 25)

- [Perpetual Humanoid Control for Real-time Simulated Avatars](https://mlsanity.com/paper/2305.06456/) (2023)
- [Physically Plausible Full-Body Hand-Object Interaction Synthesis](https://mlsanity.com/paper/2309.07907/) (2023)
- [Uplift and Upsample: Efficient 3D Human Pose Estimation with Uplifting Transformers](https://mlsanity.com/paper/2210.06110/) (2022)
- [Attention Diversification for Domain Generalization](https://mlsanity.com/paper/2210.04206/) (2022)
- [Embodied Scene-aware Human Pose Estimation](https://mlsanity.com/paper/2206.09106/) (2022)
- [Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions](https://mlsanity.com/paper/2410.12773/) (2024)
- [Global Adaptation meets Local Generalization: Unsupervised Domain Adaptation for 3D Human Pose Estimation](https://mlsanity.com/paper/2303.16456/) (2023)
- [PhysReaction: Physically Plausible Real-Time Humanoid Reaction Synthesis via Forward Dynamics Guided 4D Imitation](https://mlsanity.com/paper/2404.01081/) (2024)
- [Real-Time Simulated Avatar from Head-Mounted Sensors](https://mlsanity.com/paper/2403.06862/) (2024)
- [Hallucination Improves the Performance of Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2307.12168/) (2023)
- [Contact-aware Human Motion Generation from Textual Descriptions](https://mlsanity.com/paper/2403.15709/) (2024)
- [3D Pose Based Feedback for Physical Exercises](https://mlsanity.com/paper/2208.03257/) (2022)
- [CP3: Channel Pruning Plug-in for Point-based Networks](https://mlsanity.com/paper/2303.13097/) (2023)
- [MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos](https://mlsanity.com/paper/2512.10881/) (2025)
- [ProxyCap: Real-time Monocular Full-body Capture in World Space via Human-Centric Proxy-to-Motion Learning](https://mlsanity.com/paper/2307.01200/) (2023)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)
- [A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motions](https://mlsanity.com/paper/2412.17377/) (2024)
- [Skeleton2Humanoid: Animating Simulated Characters for Physically-plausible Motion In-betweening](https://mlsanity.com/paper/2210.04294/) (2022)
- [GraMMaR: Ground-aware Motion Model for 3D Human Motion Reconstruction](https://mlsanity.com/paper/2306.16736/) (2023)
- [LightHouse: A Survey of AGI Hallucination](https://mlsanity.com/paper/2401.06792/) (2024)
- [Physics-Guided Human Motion Capture with Pose Probability Modeling](https://mlsanity.com/paper/2308.09910/) (2023)
- [Exploring Latent Cross-Channel Embedding for Accurate 3D Human Pose Reconstruction in a Diffusion Framework](https://mlsanity.com/paper/2401.09836/) (2024)
- [Advancing Monocular Video-Based Gait Analysis Using Motion Imitation with Physics-Based Simulation](https://mlsanity.com/paper/2402.12676/) (2024)
- [Every Angle Is Worth A Second Glance: Mining Kinematic Skeletal Structures from Multi-view Joint Cloud](https://mlsanity.com/paper/2502.02936/) (2025)
- [Semantics-aware Test-time Adaptation for 3D Human Pose Estimation](https://mlsanity.com/paper/2502.10724/) (2025)

## Abstract (arXiv)

Existing self-supervised 3D human pose estimation schemes have largely relied on weak supervisions like consistency loss to guide the learning, which, inevitably, leads to inferior results in real-world scenarios with unseen poses. In this paper, we propose a novel self-supervised approach that allows us to explicitly generate 2D-3D pose pairs for augmenting supervision, through a self-enhancing dual-loop learning framework. This is made possible via introducing a reinforcement-learning-based imitator, which is learned jointly with a pose estimator alongside a pose hallucinator; the three components form two loops during the training process, complementing and strengthening one another. Specifically, the pose estimator transforms an input 2D pose sequence to a low-fidelity 3D output, which is then enhanced by the imitator that enforces physical constraints. The refined 3D poses are subsequently fed to the hallucinator for producing even more diverse data, which are, in turn, strengthened by the imitator and further utilized to train the pose estimator. Such a co-evolution scheme, in practice, enables training a pose estimator on self-generated motion data without relying on any given 3D data. Extensive experiments across various benchmarks demonstrate that our approach yields encouraging results significantly outperforming the state of the art and, in some cases, even on par with results of fully-supervised methods. Notably, it achieves 89.1% 3D PCK on MPI-INF-3DHP under self-supervised cross-dataset evaluation setup, improving upon the previous best self-supervised methods by 8.6%. Code can be found at: https://github.com/Garfield-kh/PoseTriplet