mlsanityData to 14 Oct 2026

MPT: Mesh Pre-Training with Transformers for Human Pose and Mesh Reconstruction

arXiv 2211.13357 ↗First posted 24 Nov 2022cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Traditional methods of reconstructing 3D human pose and mesh from single images rely on paired image-mesh datasets, which can be difficult and expensive to obtain. Due to this limitation, model scalability is constrained as well as reconstruction performance. Towards addressing the challenge, we introduce Mesh Pre-Training (MPT), an effective pre-training strategy that leverages large amounts of MoCap data to effectively perform pre-training at scale. We introduce the use of MoCap-generated heatmaps as input representations to the mesh regression transformer and propose a Masked Heatmap Modeling approach for improving pre-training performance. This study demonstrates that pre-training using the proposed MPT allows our models to perform effective inference without requiring fine-tuning. We further show that fine-tuning the pre-trained MPT model considerably improves the accuracy of human mesh reconstruction from single images. Experimental results show that MPT outperforms previous state-of-the-art methods on Human3.6M and 3DPW datasets. As a further application, we benchmark and study MPT on the task of 3D hand reconstruction, showing that our generic pre-training scheme generalizes well to hand pose estimation and achieves promising reconstruction performance.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 64 of its 84 references that are in the map. First posted on arXiv 24 Nov 2022.

Builds on

64 of 84 references.

References · 64

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  8. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  9. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  10. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  12. 2022Human Motion Diffusion Modelcited by 1,080
38 more
  1. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
  2. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 915
  3. 2017End-to-end Recovery of Human Shape and Posecited by 885
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  5. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 627
  6. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 620
  7. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 575
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  9. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  10. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
  11. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 418
  12. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 408
  13. 2017A simple yet effective baseline for 3d human pose estimationcited by 401
  14. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  15. 2017Learning from Synthetic Humanscited by 359
  16. 2017Cascaded Pyramid Network for Multi-Person Pose Estimationcited by 341
  17. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 337
  18. 2018Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomizationcited by 294
  19. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  20. 20213D Human Pose Estimation with Spatial and Temporal Transformerscited by 272
  21. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  22. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  23. 2021Mesh Graphormercited by 265
  24. 2021PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loopcited by 250
  25. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  26. 2019HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimationcited by 247
  27. 2022CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimationcited by 238
  28. 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
  29. 2020Monocular, One-stage, Regression of Multiple 3D Peoplecited by 218
  30. 2016Associative Embedding: End-to-End Learning for Joint Detection and Groupingcited by 212
  31. 2017Unite the People: Closing the Loop Between 3D and 2D Human Representationscited by 205
  32. 2017Single-Shot Multi-Person 3D Pose Estimation From Monocular RGBcited by 195
  33. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  34. 2020Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Videocited by 163
  35. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  36. 2020Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimationcited by 113
  37. 2022Recovering 3D Human Mesh from Monocular Images: A Surveycited by 103
  38. 2017Self-supervised Learning of Motion Capturecited by 97

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 7; 2025: 1.

11 papers.

  1. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  2. 2023HuRef: HUman-REadable Fingerprint for Large Language Modelscited by 34
  3. 2024MEGA: Masked Generative Autoencoder for Human Mesh Recoverycited by 14
  4. 2024Pre-training a Density-Aware Pose Transformer for Robust LiDAR-based 3D Human Pose Estimationcited by 5
  5. 2024MoCap-to-Visual Domain Adaptation for Efficient Human Mesh Estimation from 2D Keypointscited by 3
  6. 2024PostoMETRO: Pose Token Enhanced Mesh Transformer for Robust 3D Human Mesh Recoverycited by 2
  7. 2024Incorporating Test-Time Optimization into Training with Dual Networks for Human Mesh Recoverycited by 1
  8. 2023Monocular 3D Hand Mesh Recovery via Dual Noise Estimationcited by 0
  9. 2024Multi-RoI Human Mesh Recovery with Camera Consistency and Contrastive Lossescited by 0
  10. 2024Mesh Denoising Transformercited by 0
  11. 2025VOccl3D: A Video Benchmark Dataset for 3D Human Pose and Shape Estimation under real Occlusionscited by 0