# MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild

arXiv: https://arxiv.org/abs/1607.02046 · HTML: https://mlsanity.com/paper/1607.02046/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2016.

## Built on by (most cited first, 30 of 37)

- [BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion](https://mlsanity.com/paper/2306.16940/) (2023)
- [PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation](https://mlsanity.com/paper/2105.02465/) (2021)
- [SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation](https://mlsanity.com/paper/2304.10417/) (2023)
- [Human-Aware Object Placement for Visual Environment Reconstruction](https://mlsanity.com/paper/2203.03609/) (2022)
- [Recent Advances in Monocular 2D and 3D Human Pose Estimation: A Deep Learning Perspective](https://mlsanity.com/paper/2104.11536/) (2021)
- [A survey of synthetic data augmentation methods in computer vision](https://mlsanity.com/paper/2403.10075/) (2024)
- [AdaptPose: Cross-Dataset Adaptation for 3D Human Pose Estimation by Learnable Motion Generation](https://mlsanity.com/paper/2112.11593/) (2021)
- [RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Stream](https://mlsanity.com/paper/2104.14837/) (2021)
- [Revitalizing Optimization for 3D Human Pose and Shape Estimation: A Sparse Constrained Formulation](https://mlsanity.com/paper/2105.13965/) (2021)
- [Global Adaptation meets Local Generalization: Unsupervised Domain Adaptation for 3D Human Pose Estimation](https://mlsanity.com/paper/2303.16456/) (2023)
- [VQ-HPS: Human Pose and Shape Estimation in a Vector-Quantized Latent Space](https://mlsanity.com/paper/2312.08291/) (2023)
- [Leveraging MoCap Data for Human Mesh Recovery](https://mlsanity.com/paper/2110.09243/) (2021)
- [PoSynDA: Multi-Hypothesis Pose Synthesis Domain Adaptation for Robust 3D Human Pose Estimation](https://mlsanity.com/paper/2308.09678/) (2023)
- [MPT: Mesh Pre-Training with Transformers for Human Pose and Mesh Reconstruction](https://mlsanity.com/paper/2211.13357/) (2022)
- [Human Image Generation: A Comprehensive Survey](https://mlsanity.com/paper/2212.08896/) (2022)
- [Self-supervised Human Mesh Recovery with Cross-Representation Alignment](https://mlsanity.com/paper/2209.04596/) (2022)
- [Progressive Multi-view Human Mesh Recovery with Self-Supervision](https://mlsanity.com/paper/2212.05223/) (2022)
- [DH-AUG: DH Forward Kinematics Model Driven Augmentation for 3D Human Pose Estimation](https://mlsanity.com/paper/2207.09303/) (2022)
- [Weakly-supervised Pre-training for 3D Human Pose Estimation via Perspective Knowledge](https://mlsanity.com/paper/2211.11983/) (2022)
- [CameraPose: Weakly-Supervised Monocular 3D Human Pose Estimation by Leveraging In-the-wild 2D Annotations](https://mlsanity.com/paper/2301.02979/) (2023)
- [Diversifying Human Pose in Synthetic Data for Aerial-view Human Detection](https://mlsanity.com/paper/2405.15939/) (2024)
- [PoseGU: 3D Human Pose Estimation with Novel Human Pose Generator and Unbiased Learning](https://mlsanity.com/paper/2207.03618/) (2022)
- [WorldGen: A Large Scale Generative Simulator](https://mlsanity.com/paper/2210.00715/) (2022)
- [Skeleton-Graph: Long-Term 3D Motion Prediction From 2D Observations Using Deep Spatio-Temporal Graph CNNs](https://mlsanity.com/paper/2109.10257/) (2021)
- [PoseGen: Learning to Generate 3D Human Pose Dataset with NeRF](https://mlsanity.com/paper/2312.14915/) (2023)
- [Data Augmentation in Human-Centric Vision](https://mlsanity.com/paper/2403.08650/) (2024)
- [Di2Pose: Discrete Diffusion Model for Occluded 3D Human Pose Estimation](https://mlsanity.com/paper/2405.17016/) (2024)
- [Nuisance-Label Supervision: Robustness Improvement by Free Labels](https://mlsanity.com/paper/2110.07118/) (2021)
- [CLR-GAM: Contrastive Point Cloud Learning with Guided Augmentation and Feature Mapping](https://mlsanity.com/paper/2302.14306/) (2023)
- [Gait Data Augmentation using Physics-Based Biomechanical Simulation](https://mlsanity.com/paper/2307.08092/) (2023)

## Abstract (arXiv)

This paper addresses the problem of 3D human pose estimation in the wild. A significant challenge is the lack of training data, i.e., 2D images of humans annotated with 3D poses. Such data is necessary to train state-of-the-art CNN architectures. Here, we propose a solution to generate a large set of photorealistic synthetic images of humans with 3D pose annotations. We introduce an image-based synthesis engine that artificially augments a dataset of real images with 2D human pose annotations using 3D Motion Capture (MoCap) data. Given a candidate 3D pose our algorithm selects for each joint an image whose 2D pose locally matches the projected 3D pose. The selected images are then combined to generate a new synthetic image by stitching local image patches in a kinematically constrained manner. The resulting images are used to train an end-to-end CNN for full-body 3D pose estimation. We cluster the training data into a large number of pose classes and tackle pose estimation as a K-way classification problem. Such an approach is viable only with large training sets such as ours. Our method outperforms the state of the art in terms of 3D pose estimation in controlled environments (Human3.6M) and shows promising results for in-the-wild images (LSP). This demonstrates that CNNs trained on artificial images generalize well to real images.