# Fill in Fabrics: Body-Aware Self-Supervised Inpainting for Image-Based Virtual Try-On

arXiv: https://arxiv.org/abs/2210.00918 · HTML: https://mlsanity.com/paper/2210.00918/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Road Extraction by Deep Residual U-Net](https://mlsanity.com/paper/1711.10684/) (2017)
- [VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization](https://mlsanity.com/paper/2103.16874/) (2021)
- [VITON: An Image-based Virtual Try-on Network](https://mlsanity.com/paper/1711.08447/) (2017)
- [Toward Characteristic-Preserving Image-based Virtual Try-On Network](https://mlsanity.com/paper/1807.07688/) (2018)
- [Self-Correction for Human Parsing](https://mlsanity.com/paper/1910.09777/) (2019)
- [Towards Photo-Realistic Virtual Try-On by Adaptively GeneratingleftrightarrowPreserving Image Content](https://mlsanity.com/paper/2003.05863/) (2020)
- [Disentangled Cycle Consistency for Highly-realistic Virtual Try-On](https://mlsanity.com/paper/2103.09479/) (2021)
- [The Conditional Analogy GAN: Swapping Fashion Articles on People Images](https://mlsanity.com/paper/1709.04695/) (2017)
- [SieveNet: A Unified Framework for Robust Image-Based Virtual Try-On](https://mlsanity.com/paper/2001.06265/) (2020)
- [Cloth Interactive Transformer for Virtual Try-On](https://mlsanity.com/paper/2104.05519/) (2021)

## Built on by (most cited first, 1 of 1)

- [Image-Based Virtual Try-On: A Survey](https://mlsanity.com/paper/2311.04811/) (2023)

## Abstract (arXiv)

Previous virtual try-on methods usually focus on aligning a clothing item with a person, limiting their ability to exploit the complex pose, shape and skin color of the person, as well as the overall structure of the clothing, which is vital to photo-realistic virtual try-on. To address this potential weakness, we propose a fill in fabrics (FIFA) model, a self-supervised conditional generative adversarial network based framework comprised of a Fabricator and a unified virtual try-on pipeline with a Segmenter, Warper and Fuser. The Fabricator aims to reconstruct the clothing image when provided with a masked clothing as input, and learns the overall structure of the clothing by filling in fabrics. A virtual try-on pipeline is then trained by transferring the learned representations from the Fabricator to Warper in an effort to warp and refine the target clothing. We also propose to use a multi-scale structural constraint to enforce global context at multiple scales while warping the target clothing to better fit the pose and shape of the person. Extensive experiments demonstrate that our FIFA model achieves state-of-the-art results on the standard VITON dataset for virtual try-on of clothing items, and is shown to be effective at handling complex poses and retaining the texture and embroidery of the clothing.