# THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers

arXiv: https://arxiv.org/abs/2106.09336 · HTML: https://mlsanity.com/paper/2106.09336/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [Unite the People: Closing the Loop Between 3D and 2D Human Representations](https://mlsanity.com/paper/1701.02468/) (2017)
- [Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB](https://mlsanity.com/paper/1712.03453/) (2017)
- [Monocular Expressive Body Regression through Body-Driven Attention](https://mlsanity.com/paper/2008.09062/) (2020)
- [BodyNet: Volumetric Inference of 3D Human Body Shapes](https://mlsanity.com/paper/1804.04875/) (2018)
- [Coherent Reconstruction of Multiple Humans from a Single Image](https://mlsanity.com/paper/2006.08586/) (2020)
- [VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera](https://mlsanity.com/paper/1705.01583/) (2017)
- [We are More than Our Joints: Predicting how 3D Bodies Move](https://mlsanity.com/paper/2012.00619/) (2020)
- [Exploiting temporal context for 3D human pose estimation in the wild](https://mlsanity.com/paper/1905.04266/) (2019)
- [DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare](https://mlsanity.com/paper/1910.00116/) (2019)
- [Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flows](https://mlsanity.com/paper/2003.10350/) (2020)
- [Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation](https://mlsanity.com/paper/1804.01110/) (2018)
- [Hierarchical Kinematic Human Mesh Recovery](https://mlsanity.com/paper/2003.04232/) (2020)
- [Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild](https://mlsanity.com/paper/2003.07581/) (2020)
- [3D Human Mesh Regression with Dense Correspondence](https://mlsanity.com/paper/2006.05734/) (2020)
- [Neural Descent for Visual 3D Human Pose and Shape](https://mlsanity.com/paper/2008.06910/) (2020)

## Built on by (most cited first, 30 of 50)

- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion](https://mlsanity.com/paper/2306.16940/) (2023)
- [GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras](https://mlsanity.com/paper/2112.01524/) (2021)
- [PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images](https://mlsanity.com/paper/2207.06400/) (2022)
- [GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping](https://mlsanity.com/paper/2112.11454/) (2021)
- [Recovering 3D Human Mesh from Monocular Images: A Survey](https://mlsanity.com/paper/2203.01923/) (2022)
- [Human Motion Generation: A Survey](https://mlsanity.com/paper/2307.10894/) (2023)
- [Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2407.07532/) (2024)
- [FLAG: Flow-based 3D Avatar Generation from Sparse Observations](https://mlsanity.com/paper/2203.05789/) (2022)
- [3D Human Mesh Estimation from Virtual Markers](https://mlsanity.com/paper/2303.11726/) (2023)
- [Differentiable Dynamics for Articulated 3d Human Motion Reconstruction](https://mlsanity.com/paper/2205.12256/) (2022)
- [Motion Flow Matching for Human Motion Synthesis and Editing](https://mlsanity.com/paper/2312.08895/) (2023)
- [PACE: Human and Camera Motion Estimation from in-the-wild Videos](https://mlsanity.com/paper/2310.13768/) (2023)
- [PSVT: End-to-End Multi-person 3D Pose and Shape Estimation with Progressive Video Transformers](https://mlsanity.com/paper/2303.09187/) (2023)
- [POTTER: Pooling Attention Transformer for Efficient Human Mesh Recovery](https://mlsanity.com/paper/2303.13357/) (2023)
- [HSPACE: Synthetic Parametric Humans Animated in Complex Environments](https://mlsanity.com/paper/2112.12867/) (2021)
- [Benchmarking and Analyzing 3D Human Pose and Shape Estimation Beyond Algorithms](https://mlsanity.com/paper/2209.10529/) (2022)
- [HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations](https://mlsanity.com/paper/2403.03561/) (2024)
- [HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving](https://mlsanity.com/paper/2212.07729/) (2022)
- [3D Vision with Transformers: A Survey](https://mlsanity.com/paper/2208.04309/) (2022)
- [POCO: 3D Pose and Shape Estimation with Confidence](https://mlsanity.com/paper/2308.12965/) (2023)
- [Structured 3D Features for Reconstructing Controllable Avatars](https://mlsanity.com/paper/2212.06820/) (2022)
- [Scene-Aware 3D Multi-Human Motion Capture from a Single Camera](https://mlsanity.com/paper/2301.05175/) (2023)
- [HMD-NeMo: Online 3D Avatar Motion Generation From Sparse Observations](https://mlsanity.com/paper/2308.11261/) (2023)
- [COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimation](https://mlsanity.com/paper/2408.16426/) (2024)
- [Reconstructing Three-Dimensional Models of Interacting Humans](https://mlsanity.com/paper/2308.01854/) (2023)
- [BlazePose GHUM Holistic: Real-time 3D Human Landmarks and Pose Estimation](https://mlsanity.com/paper/2206.11678/) (2022)
- [DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans](https://mlsanity.com/paper/2404.00485/) (2024)
- [MPT: Mesh Pre-Training with Transformers for Human Pose and Mesh Reconstruction](https://mlsanity.com/paper/2211.13357/) (2022)
- [FeatER: An Efficient Network for Human Reconstruction via Feature Map-Based TransformER](https://mlsanity.com/paper/2205.15448/) (2022)

## Abstract (arXiv)

We present THUNDR, a transformer-based deep neural network methodology to reconstruct the 3d pose and shape of people, given monocular RGB images. Key to our methodology is an intermediate 3d marker representation, where we aim to combine the predictive power of model-free-output architectures and the regularizing, anthropometrically-preserving properties of a statistical human surface model like GHUM -- a recently introduced, expressive full body statistical 3d human model, trained end-to-end. Our novel transformer-based prediction pipeline can focus on image regions relevant to the task, supports self-supervised regimes, and ensures that solutions are consistent with human anthropometry. We show state-of-the-art results on Human3.6M and 3DPW, for both the fully-supervised and the self-supervised models, for the task of inferring 3d human shape, joint positions, and global translation. Moreover, we observe very solid 3d reconstruction performance for difficult human poses collected in the wild.