mlsanityData to 7 Oct 2026

Leveraging MoCap Data for Human Mesh Recovery

arXiv 2110.09243 ↗First posted 18 Oct 2021cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Training state-of-the-art models for human body pose and shape recovery from images or videos requires datasets with corresponding annotations that are really hard and expensive to obtain. Our goal in this paper is to study whether poses from 3D Motion Capture (MoCap) data can be used to improve image-based and video-based human mesh recovery methods. We find that fine-tune image-based models with synthetic renderings from MoCap data can increase their performance, by providing them with a wider variety of poses, textures and backgrounds. In fact, we show that simply fine-tuning the batch normalization layers of the model is enough to achieve large gains. We further study the use of MoCap data for video, and introduce PoseBERT, a transformer module that directly regresses the pose parameters and is trained via masked modeling. It is simple, generic and can be plugged on top of any state-of-the-art image-based model in order to transform it in a video-based model leveraging temporal information. Our experimental results show that the proposed approaches reach state-of-the-art performance on various datasets including 3DPW, MPI-INF-3DHP, MuPoTS-3D, MCB and AIST. Test code and models will be available soon.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 42 of its 50 references that are in the map. First posted on arXiv 18 Oct 2021.

Builds on

42 of 50 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  8. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  9. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  10. 2017End-to-end Recovery of Human Shape and Posecited by 883
  11. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  12. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
30 more
  1. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  2. 2016Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervisioncited by 493
  3. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  4. 2017Learning from Synthetic Humanscited by 358
  5. 2018Learning 3D Human Dynamics from Videocited by 315
  6. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  7. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  8. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  9. 2020Monocular, One-stage, Regression of Multiple 3D Peoplecited by 216
  10. 2017Unite the People: Closing the Loop Between 3D and 2D Human Representationscited by 205
  11. 2017Single-Shot Multi-Person 3D Pose Estimation From Monocular RGBcited by 195
  12. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  13. 2020Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Videocited by 162
  14. 2018BodyNet: Volumetric Inference of 3D Human Body Shapescited by 141
  15. 2019Exploiting temporal context for 3D human pose estimation in the wildcited by 110
  16. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 103
  17. 20203D Human Motion Estimation via Motion Compression and Refinementcited by 98
  18. 2017Towards Accurate Markerless Human Shape and Pose Estimation over Timecited by 92
  19. 2018LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Imagescited by 88
  20. 2019Human Mesh Recovery from Monocular Images via a Skeleton-disentangled Representationcited by 87
  21. 2019Sim2real transfer learning for 3D human pose estimation: motion to the rescuecited by 75
  22. 2019TexturePose: Supervising Human Mesh Estimation with Texture Consistencycited by 53
  23. 2016Synthesizing Training Images for Boosting Human 3D Pose Estimationcited by 52
  24. 2020Human Mesh Recovery from Multiple Shotscited by 52
  25. 2019Delving Deep Into Hybrid Annotations for 3D Human Recovery in the Wildcited by 38
  26. 2016MoCap-guided Data Augmentation for 3D Pose Estimation in the Wildcited by 37
  27. 2021Skeletor: Skeletal Transformers for Robust Body-Pose Estimationcited by 26
  28. 2020SimPose: Effectively Learning DensePose and Surface Normals of People from Simulated Datacited by 17
  29. 2020SMPLy Benchmarking 3D Human Pose Estimation in the Wildcited by 16
  30. 2020Unsupervised Cross-Modal Alignment for Multi-Person 3D Pose Estimationcited by 10

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 3; 2024: 5; 2025: 1.

12 papers.

  1. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 344
  2. 2022PoseGPT: Quantization-based 3D Human Motion Generation and Forecastingcited by 78
  3. 2023On the Benefits of 3D Pose and Tracking for Human Action Recognitioncited by 32
  4. 2022PoseBERT: A Generic Transformer Module for Temporal 3D Human Modelingcited by 15
  5. 2024MEGA: Masked Generative Autoencoder for Human Mesh Recoverycited by 13
  6. 2024MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognitioncited by 13
  7. 2022MPT: Mesh Pre-Training with Transformers for Human Pose and Mesh Reconstructioncited by 11
  8. 2023Rethinking Self-Supervised Visual Representation Learning in Pre-training for 3D Human Pose and Shape Estimationcited by 7
  9. 2025HumanMM: Global Human Motion Recovery from Multi-shot Videoscited by 4
  10. 2024Multi-Human Mesh Recovery with Transformerscited by 1
  11. 2024HPE-CogVLM: Advancing Vision Language Models with a Head Pose Grounding Taskcited by 1
  12. 2024Temporally-consistent 3D Reconstruction of Birdscited by 1