# ControlEdit: A MultiModal Local Clothing Image Editing Method

arXiv: https://arxiv.org/abs/2409.14720 · HTML: https://mlsanity.com/paper/2409.14720/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 28 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [Scaling up GANs for Text-to-Image Synthesis](https://mlsanity.com/paper/2303.05511/) (2023)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtion](https://mlsanity.com/paper/2111.12417/) (2021)
- [Controllable Person Image Synthesis with Attribute-Decomposed GAN](https://mlsanity.com/paper/2003.12267/) (2020)
- [ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts](https://mlsanity.com/paper/2210.15257/) (2022)
- [Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editing](https://mlsanity.com/paper/2304.02051/) (2023)
- [Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model](https://mlsanity.com/paper/2310.07222/) (2023)
- [Be Your Own Prada: Fashion Synthesis with Structural Coherence](https://mlsanity.com/paper/1710.07346/) (2017)
- [Designing a Better Asymmetric VQGAN for StableDiffusion](https://mlsanity.com/paper/2306.04632/) (2023)
- [ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation](https://mlsanity.com/paper/2112.15283/) (2021)
- [DiffFashion: Reference-based Fashion Design with Structure-aware Transfer by Diffusion Models](https://mlsanity.com/paper/2302.06826/) (2023)
- [FICE: Text-Conditioned Fashion Image Editing With Guided GAN Inversion](https://mlsanity.com/paper/2301.02110/) (2023)
- [Fashion++: Minimal Edits for Outfit Improvement](https://mlsanity.com/paper/1904.09261/) (2019)
- [Reference-based Image Composition with Sketch via Structure-aware Diffusion Model](https://mlsanity.com/paper/2304.09748/) (2023)
- [FashionTex: Controllable Virtual Try-on with Text and Texture](https://mlsanity.com/paper/2305.04451/) (2023)

## Abstract (arXiv)

Multimodal clothing image editing refers to the precise adjustment and modification of clothing images using data such as textual descriptions and visual images as control conditions, which effectively improves the work efficiency of designers and reduces the threshold for user design. In this paper, we propose a new image editing method ControlEdit, which transfers clothing image editing to multimodal-guided local inpainting of clothing images. We address the difficulty of collecting real image datasets by leveraging the self-supervised learning approach. Based on this learning approach, we extend the channels of the feature extraction network to ensure consistent clothing image style before and after editing, and we design an inverse latent loss function to achieve soft control over the content of non-edited areas. In addition, we adopt Blended Latent Diffusion as the sampling method to make the editing boundaries transition naturally and enforce consistency of non-edited area content. Extensive experiments demonstrate that ControlEdit surpasses baseline algorithms in both qualitative and quantitative evaluations.