# Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model

arXiv: https://arxiv.org/abs/2111.13333 · HTML: https://mlsanity.com/paper/2111.13333/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 44 of its 51 references that are in the map.
- First posted on arXiv 26 Nov 2021.

## Placed in

- Multimodal LLMs / vision-language models / [Image editing and quality assessment](https://mlsanity.com/mllm/image-editing-and-quality-assessment/)

## Builds on (most cited first, 30 of 44)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [Pivotal Tuning for Latent-based Editing of Real Images](https://mlsanity.com/paper/2106.05744/) (2021)
- [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows](https://mlsanity.com/paper/2008.02401/) (2020)
- [StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation](https://mlsanity.com/paper/2011.12799/) (2020)
- [Unsupervised Discovery of Interpretable Directions in the GAN Latent Space](https://mlsanity.com/paper/2002.03754/) (2020)
- [StyleRig: Rigging StyleGAN for 3D Control over Portrait Images](https://mlsanity.com/paper/2004.00121/) (2020)
- [On the "steerability" of generative adversarial networks](https://mlsanity.com/paper/1907.07171/) (2019)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators](https://mlsanity.com/paper/2108.00946/) (2021)
- [ManiGAN: Text-Guided Image Manipulation](https://mlsanity.com/paper/1912.06203/) (2019)
- [VinVL: Revisiting Visual Representations in Vision-Language Models](https://mlsanity.com/paper/2101.00529/) (2021)

## Built on by (most cited first, 30 of 34)

- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [Diffusion Models already have a Semantic Latent Space](https://mlsanity.com/paper/2210.10960/) (2022)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis](https://mlsanity.com/paper/2301.12959/) (2023)
- [Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation](https://mlsanity.com/paper/2312.10113/) (2023)
- [Shape-aware Text-driven Layered Video Editing](https://mlsanity.com/paper/2301.13173/) (2023)
- [Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation](https://mlsanity.com/paper/2206.07771/) (2022)
- [DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulation](https://mlsanity.com/paper/2303.06285/) (2023)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [CLIP-PAE: Projection-Augmentation Embedding to Extract Relevant Features for a Disentangled, Interpretable, and Controllable Text-Guided Face Manipulation](https://mlsanity.com/paper/2210.03919/) (2022)
- [Not Only Generative Art: Stable Diffusion for Content-Style Disentanglement in Art Analysis](https://mlsanity.com/paper/2304.10278/) (2023)
- [clip2latent: Text driven sampling of a pre-trained StyleGAN using denoising diffusion and CLIP](https://mlsanity.com/paper/2210.02347/) (2022)
- [Zero-Shot Text-to-Parameter Translation for Game Character Auto-Creation](https://mlsanity.com/paper/2303.01311/) (2023)
- [Eliminating Contextual Prior Bias for Semantic Image Editing via Dual-Cycle Diffusion](https://mlsanity.com/paper/2302.02394/) (2023)
- [DE-Net: Dynamic Text-guided Image Editing Adversarial Networks](https://mlsanity.com/paper/2206.01160/) (2022)
- [Text-guided Eyeglasses Manipulation with Spatial Constraints](https://mlsanity.com/paper/2304.12539/) (2023)
- [Bipartite Graph Reasoning GANs for Person Pose and Facial Image Synthesis](https://mlsanity.com/paper/2211.06719/) (2022)
- [FashionTex: Controllable Virtual Try-on with Text and Texture](https://mlsanity.com/paper/2305.04451/) (2023)
- [Closed-Loop Unsupervised Representation Disentanglement with β-VAE Distillation and Diffusion Probabilistic Feedback](https://mlsanity.com/paper/2402.02346/) (2024)
- [Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesis](https://mlsanity.com/paper/2307.12084/) (2023)
- [End-to-End Visual Editing with a Generatively Pre-Trained Artist](https://mlsanity.com/paper/2205.01668/) (2022)
- [Text-driven Visual Synthesis with Latent Diffusion Prior](https://mlsanity.com/paper/2302.08510/) (2023)
- [DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learning](https://mlsanity.com/paper/2408.07080/) (2024)
- [Graph Transformer GANs with Graph Masked Modeling for Architectural Layout Generation](https://mlsanity.com/paper/2401.07721/) (2024)
- [A Survey of Multimodal Composite Editing and Retrieval](https://mlsanity.com/paper/2409.05405/) (2024)
- [Facial Expression Translation using Landmark Guided GANs](https://mlsanity.com/paper/2209.02136/) (2022)
- [SpectralCLIP: Preventing Artifacts in Text-Guided Style Transfer from a Spectral Perspective](https://mlsanity.com/paper/2303.09270/) (2023)
- [DeltaSpace: A Semantic-aligned Feature Space for Flexible Text-guided Image Editing](https://mlsanity.com/paper/2310.08785/) (2023)
- [Identity-Preserving Text-to-Image Generation via Dual-Level Feature Decoupling and Expert-Guided Fusion](https://mlsanity.com/paper/2505.22360/) (2025)

## Abstract (arXiv)

To achieve disentangled image manipulation, previous works depend heavily on manual annotation. Meanwhile, the available manipulations are limited to a pre-defined set the models were trained for. We propose a novel framework, i.e., Predict, Prevent, and Evaluate (PPE), for disentangled text-driven image manipulation that requires little manual annotation while being applicable to a wide variety of manipulations. Our method approaches the targets by deeply exploiting the power of the large-scale pre-trained vision-language model CLIP. Concretely, we firstly Predict the possibly entangled attributes for a given text command. Then, based on the predicted attributes, we introduce an entanglement loss to Prevent entanglements during training. Finally, we propose a new evaluation metric to Evaluate the disentangled image manipulation. We verify the effectiveness of our method on the challenging face editing task. Extensive experiments show that the proposed PPE framework achieves much better quantitative and qualitative results than the up-to-date StyleCLIP baseline.