# Co-Evolution of Pose and Mesh for 3D Human Body Estimation from Video

arXiv: https://arxiv.org/abs/2308.10305 · HTML: https://mlsanity.com/paper/2308.10305/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 57 references that are in the map.
- First posted on arXiv 20 Aug 2023.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2011.14672/) (2020)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [3D Human Pose Estimation with Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.10455/) (2021)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [Mesh Graphormer](https://mlsanity.com/paper/2104.00272/) (2021)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop](https://mlsanity.com/paper/2103.16507/) (2021)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation](https://mlsanity.com/paper/2208.00571/) (2022)
- [Learning to Estimate 3D Human Pose and Shape from a Single Color Image](https://mlsanity.com/paper/1805.04092/) (2018)

## Built on by (most cited first, 25 of 25)

- [RoHM: Robust Human Motion Reconstruction via Diffusion](https://mlsanity.com/paper/2401.08570/) (2024)
- [Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning](https://mlsanity.com/paper/2404.05218/) (2024)
- [PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos](https://mlsanity.com/paper/2404.04430/) (2024)
- [3D Human Pose Perception from Egocentric Stereo Videos](https://mlsanity.com/paper/2401.00889/) (2023)
- [Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation](https://mlsanity.com/paper/2311.12028/) (2023)
- [CoMotion: Concurrent Multi-person 3D Motion](https://mlsanity.com/paper/2504.12186/) (2025)
- [Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos](https://mlsanity.com/paper/2410.07795/) (2024)
- [H2OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers](https://mlsanity.com/paper/2509.06956/) (2025)
- [PI-HMR: Towards Robust In-bed Temporal Human Shape Reconstruction with Contact Pressure Sensing](https://mlsanity.com/paper/2503.00068/) (2025)
- [Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videos](https://mlsanity.com/paper/2506.03440/) (2025)
- [PersPose: 3D Human Pose Estimation with Perspective Encoding and Perspective Rotation](https://mlsanity.com/paper/2508.17239/) (2025)
- [QuaMo: Quaternion Motions for Vision-based 3D Human Kinematics Capture](https://mlsanity.com/paper/2601.19580/) (2026)
- [FMPose3D: monocular 3D pose estimation via flow matching](https://mlsanity.com/paper/2602.05755/) (2026)
- [Uncertainty-Aware Testing-Time Optimization for 3D Human Pose Estimation](https://mlsanity.com/paper/2402.02339/) (2024)
- [HeatFormer: A Neural Optimizer for Multiview Human Mesh Recovery](https://mlsanity.com/paper/2412.04456/) (2024)
- [Masked Modeling for Human Motion Recovery Under Occlusions](https://mlsanity.com/paper/2601.16079/) (2026)
- [Task-Oriented Communication for Human Action Understanding via Edge-Cloud Co-Inference](https://mlsanity.com/paper/2605.07354/) (2026)
- [PRIMA: Boosting Animal Mesh Recovery with Biological Priors and Test-Time Adaptation](https://mlsanity.com/paper/2606.02366/) (2026)
- [Efficient Diffusion-Based 3D Human Pose Estimation with Hierarchical Temporal Pruning](https://mlsanity.com/paper/2508.21363/) (2025)
- [Hyperbolic Space Learning Method Leveraging Temporal Motion Priors for Human Mesh Recovery](https://mlsanity.com/paper/2510.18256/) (2025)
- [Latent-Info and Low-Dimensional Learning for Human Mesh Recovery and Parallel Optimization](https://mlsanity.com/paper/2510.18267/) (2025)
- [Robust Long-term Test-Time Adaptation for 3D Human Pose Estimation through Motion Discretization](https://mlsanity.com/paper/2511.18851/) (2025)
- [Towards Geometry-Aware and Motion-Guided Video Human Mesh Recovery](https://mlsanity.com/paper/2601.21376/) (2026)
- [Depth-Guided Metric-Aware Temporal Consistency for Monocular Video Human Mesh Recovery](https://mlsanity.com/paper/2602.04257/) (2026)
- [MoPO: Incorporating Motion Prior for Occluded Human Mesh Recovery](https://mlsanity.com/paper/2605.09856/) (2026)

## Abstract (arXiv)

Despite significant progress in single image-based 3D human mesh recovery, accurately and smoothly recovering 3D human motion from a video remains challenging. Existing video-based methods generally recover human mesh by estimating the complex pose and shape parameters from coupled image features, whose high complexity and low representation ability often result in inconsistent pose motion and limited shape patterns. To alleviate this issue, we introduce 3D pose as the intermediary and propose a Pose and Mesh Co-Evolution network (PMCE) that decouples this task into two parts: 1) video-based 3D human pose estimation and 2) mesh vertices regression from the estimated 3D pose and temporal image feature. Specifically, we propose a two-stream encoder that estimates mid-frame 3D pose and extracts a temporal image feature from the input image sequence. In addition, we design a co-evolution decoder that performs pose and mesh interactions with the image-guided Adaptive Layer Normalization (AdaLN) to make pose and mesh fit the human body shape. Extensive experiments demonstrate that the proposed PMCE outperforms previous state-of-the-art methods in terms of both per-frame accuracy and temporal consistency on three benchmark datasets: 3DPW, Human3.6M, and MPI-INF-3DHP. Our code is available at https://github.com/kasvii/PMCE.