# GC-VTON: Predicting Globally Consistent and Occlusion Aware Local Flows with Neighborhood Integrity Preservation for Virtual Try-on

arXiv: https://arxiv.org/abs/2311.04932 · HTML: https://mlsanity.com/paper/2311.04932/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 7 Nov 2023.

## Builds on (most cited first, 27 of 27)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [Scaling up GANs for Text-to-Image Synthesis](https://mlsanity.com/paper/2303.05511/) (2023)
- [VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization](https://mlsanity.com/paper/2103.16874/) (2021)
- [VITON: An Image-based Virtual Try-on Network](https://mlsanity.com/paper/1711.08447/) (2017)
- [Toward Characteristic-Preserving Image-based Virtual Try-On Network](https://mlsanity.com/paper/1807.07688/) (2018)
- [EditGAN: High-Precision Semantic Image Editing](https://mlsanity.com/paper/2111.03186/) (2021)
- [StyleSwin: Transformer-based GAN for High-resolution Image Generation](https://mlsanity.com/paper/2112.10762/) (2021)
- [TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style](https://mlsanity.com/paper/2003.04583/) (2020)
- [Towards Photo-Realistic Virtual Try-On by Adaptively GeneratingleftrightarrowPreserving Image Content](https://mlsanity.com/paper/2003.05863/) (2020)
- [Parser-Free Virtual Try-on via Distilling Appearance Flows](https://mlsanity.com/paper/2103.04559/) (2021)
- [Progressive Pose Attention Transfer for Person Image Generation](https://mlsanity.com/paper/1904.03349/) (2019)
- [High-Resolution Virtual Try-On with Misalignment and Occlusion-Handled Conditions](https://mlsanity.com/paper/2206.14180/) (2022)
- [Style-Based Global Appearance Flow for Virtual Try-On](https://mlsanity.com/paper/2204.01046/) (2022)
- [Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On](https://mlsanity.com/paper/2105.06462/) (2021)
- [Do Not Mask What You Do Not Need to Mask: a Parser-Free Virtual Try-On](https://mlsanity.com/paper/2007.02721/) (2020)
- [Image Restoration using Total Variation Regularized Deep Image Prior](https://mlsanity.com/paper/1810.12864/) (2018)
- [Single Stage Virtual Try-on via Deformable Attention Flows](https://mlsanity.com/paper/2207.09161/) (2022)
- [Disentangled Cycle Consistency for Highly-realistic Virtual Try-On](https://mlsanity.com/paper/2103.09479/) (2021)
- [Toward Accurate and Realistic Outfits Visualization with Attention to Details](https://mlsanity.com/paper/2106.06593/) (2021)
- [M3D-VTON: A Monocular-to-3D Virtual Try-On Network](https://mlsanity.com/paper/2108.05126/) (2021)
- [Dressing in the Wild by Watching Dance Videos](https://mlsanity.com/paper/2203.15320/) (2022)
- [CT-Net: Complementary Transfering Network for Garment Transfer with Arbitrary Geometric Changes](https://mlsanity.com/paper/2105.05497/) (2021)
- [Robust 3D Garment Digitization from Monocular 2D Images for 3D Virtual Try-On Systems](https://mlsanity.com/paper/2111.15140/) (2021)
- [Modernizing Old Photos Using Multiple References via Photorealistic Style Transfer](https://mlsanity.com/paper/2304.04461/) (2023)

## Built on by (most cited first, 1 of 1)

- [Fashionability-Enhancing Outfit Image Editing with Conditional Diffusion Models](https://mlsanity.com/paper/2412.18421/) (2024)

## Abstract (arXiv)

Flow based garment warping is an integral part of image-based virtual try-on networks. However, optimizing a single flow predicting network for simultaneous global boundary alignment and local texture preservation results in sub-optimal flow fields. Moreover, dense flows are inherently not suited to handle intricate conditions like garment occlusion by body parts or by other garments. Forcing flows to handle the above issues results in various distortions like texture squeezing, and stretching. In this work, we propose a novel approach where we disentangle the global boundary alignment and local texture preserving tasks via our GlobalNet and LocalNet modules. A consistency loss is then employed between the two modules which harmonizes the local flows with the global boundary alignment. Additionally, we explicitly handle occlusions by predicting body-parts visibility mask, which is used to mask out the occluded regions in the warped garment. The masking prevents the LocalNet from predicting flows that distort texture to compensate for occlusions. We also introduce a novel regularization loss (NIPR), that defines a criteria to identify the regions in the warped garment where texture integrity is violated (squeezed or stretched). NIPR subsequently penalizes the flow in those regions to ensure regular and coherent warps that preserve the texture in local neighborhoods. Evaluation on a widely used virtual try-on dataset demonstrates strong performance of our network compared to the current SOTA methods.