# Cloth2Body: Generating 3D Human Body Mesh from 2D Clothing arXiv: https://arxiv.org/abs/2309.16189 ยท HTML: https://mlsanity.com/paper/2309.16189/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Builds on 43 of its 57 references that are in the map. - First posted on arXiv 28 Sep 2023. ## Builds on (most cited first, 30 of 43) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013) - [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022) - [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019) - [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021) - [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017) - [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019) - [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020) - [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021) - [HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2011.14672/) (2020) - [VITON: An Image-based Virtual Try-on Network](https://mlsanity.com/paper/1711.08447/) (2017) - [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019) - [Mesh Graphormer](https://mlsanity.com/paper/2104.00272/) (2021) - [Pose Guided Person Image Generation](https://mlsanity.com/paper/1705.09368/) (2017) - [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020) - [Multi-Garment Net: Learning to Dress 3D People from Images](https://mlsanity.com/paper/1908.06903/) (2019) - [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021) - [AGORA: Avatars in Geography Optimized for Regression Analysis](https://mlsanity.com/paper/2104.14643/) (2021) - [STAR: Sparse Trained Articulated Human Body Regressor](https://mlsanity.com/paper/2008.08535/) (2020) - [Monocular Expressive Body Regression through Body-Driven Attention](https://mlsanity.com/paper/2008.09062/) (2020) - [SMPLicit: Topology-aware Generative Model for Clothed People](https://mlsanity.com/paper/2103.06871/) (2021) - [Probabilistic Modeling for Human Mesh Recovery](https://mlsanity.com/paper/2108.11944/) (2021) - [Towards Photo-Realistic Virtual Try-On by Adaptively GeneratingleftrightarrowPreserving Image Content](https://mlsanity.com/paper/2003.05863/) (2020) - [Parser-Free Virtual Try-on via Distilling Appearance Flows](https://mlsanity.com/paper/2103.04559/) (2021) - [DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Images](https://mlsanity.com/paper/1901.07973/) (2019) - [Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1907.11346/) (2019) - [Recovering 3D Human Mesh from Monocular Images: A Survey](https://mlsanity.com/paper/2203.01923/) (2022) - [Pose Recognition with Cascade Transformers](https://mlsanity.com/paper/2104.06976/) (2021) ## Built on by (most cited first, 3 of 3) - [Improving Gloss-free Sign Language Translation by Reducing Representation Density](https://mlsanity.com/paper/2405.14312/) (2024) - [GaussianVTON: 3D Human Virtual Try-ON via Multi-Stage Gaussian Splatting Editing with Image Prompting](https://mlsanity.com/paper/2405.07472/) (2024) - [ETCH: Generalizing Body Fitting to Clothed Humans via Equivariant Tightness](https://mlsanity.com/paper/2503.10624/) (2025) ## Abstract (arXiv) In this paper, we define and study a new Cloth2Body problem which has a goal of generating 3D human body meshes from a 2D clothing image. Unlike the existing human mesh recovery problem, Cloth2Body needs to address new and emerging challenges raised by the partial observation of the input and the high diversity of the output. Indeed, there are three specific challenges. First, how to locate and pose human bodies into the clothes. Second, how to effectively estimate body shapes out of various clothing types. Finally, how to generate diverse and plausible results from a 2D clothing image. To this end, we propose an end-to-end framework that can accurately estimate 3D body mesh parameterized by pose and shape from a 2D clothing image. Along this line, we first utilize Kinematics-aware Pose Estimation to estimate body pose parameters. 3D skeleton is employed as a proxy followed by an inverse kinematics module to boost the estimation accuracy. We additionally design an adaptive depth trick to align the re-projected 3D mesh better with 2D clothing image by disentangling the effects of object size and camera extrinsic. Next, we propose Physics-informed Shape Estimation to estimate body shape parameters. 3D shape parameters are predicted based on partial body measurements estimated from RGB image, which not only improves pixel-wise human-cloth alignment, but also enables flexible user editing. Finally, we design Evolution-based pose generation method, a skeleton transplanting method inspired by genetic algorithms to generate diverse reasonable poses during inference. As shown by experimental results on both synthetic and real-world data, the proposed framework achieves state-of-the-art performance and can effectively recover natural and diverse 3D body meshes from 2D images that align well with clothing.