# FlexIT: Towards Flexible Semantic Image Translation

arXiv: https://arxiv.org/abs/2203.04705 · HTML: https://mlsanity.com/paper/2203.04705/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 59 references that are in the map.
- First posted on arXiv 9 Mar 2022.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020)

## Built on by (most cited first, 28 of 28)

- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Diffusion-based Image Translation using Disentangled Style and Content Representation](https://mlsanity.com/paper/2209.15264/) (2022)
- [Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing](https://mlsanity.com/paper/2403.03431/) (2024)
- [PODA: Prompt-driven Zero-shot Domain Adaptation](https://mlsanity.com/paper/2212.03241/) (2022)
- [Region-Aware Diffusion for Zero-shot Text-driven Image Editing](https://mlsanity.com/paper/2302.11797/) (2023)
- [Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment](https://mlsanity.com/paper/2208.13628/) (2022)
- [Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks](https://mlsanity.com/paper/2401.07709/) (2024)
- [MoEController: Instruction-based Arbitrary Image Manipulation with Mixture-of-Expert Controllers](https://mlsanity.com/paper/2309.04372/) (2023)
- [FashionTex: Controllable Virtual Try-on with Text and Texture](https://mlsanity.com/paper/2305.04451/) (2023)
- [Text-guided Explorable Image Super-resolution](https://mlsanity.com/paper/2403.01124/) (2024)
- [Towards Deeper Emotional Reflection: Crafting Affective Image Filters with Generative Priors](https://mlsanity.com/paper/2512.17376/) (2025)
- [Bridging Day and Night: Target-Class Hallucination Suppression in Unpaired Image Translation](https://mlsanity.com/paper/2602.15383/) (2026)
- [Controllable Image Generation via Collage Representations](https://mlsanity.com/paper/2304.13722/) (2023)
- [Text-to-Image Cross-Modal Generation: A Systematic Review](https://mlsanity.com/paper/2401.11631/) (2024)
- [Spatial-temporal Transformer-guided Diffusion based Data Augmentation for Efficient Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.13434/) (2023)
- [Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval](https://mlsanity.com/paper/2212.04267/) (2022)
- [Design Booster: A Text-Guided Diffusion Model for Image Translation with Spatial Layout Preservation](https://mlsanity.com/paper/2302.02284/) (2023)
- [Improving Diffusion-based Image Translation using Asymmetric Gradient Guidance](https://mlsanity.com/paper/2306.04396/) (2023)
- [Exploring the Potentials and Challenges of Deep Generative Models in Product Design Conception](https://mlsanity.com/paper/2407.11104/) (2024)
- [CSHNet: A Novel Information Asymmetric Image Translation Method](https://mlsanity.com/paper/2501.10197/) (2025)
- [Prmpt2Adpt: Prompt-Based Zero-Shot Domain Adaptation for Resource-Constrained Environments](https://mlsanity.com/paper/2506.16994/) (2025)
- [FBSDiff++: Improved Frequency Band Substitution of Diffusion Features for Efficient and Highly Controllable Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2601.19115/) (2026)
- [Automatic Geo-alignment of Artwork in Children's Story Books](https://mlsanity.com/paper/2304.01204/) (2023)
- [DM-Align: Leveraging the Power of Natural Language Instructions to Make Changes to Images](https://mlsanity.com/paper/2404.18020/) (2024)
- [Learning Latent Representations for Image Translation using Frequency Distributed CycleGAN](https://mlsanity.com/paper/2508.03415/) (2025)

## Abstract (arXiv)

Deep generative models, like GANs, have considerably improved the state of the art in image synthesis, and are able to generate near photo-realistic images in structured domains such as human faces. Based on this success, recent work on image editing proceeds by projecting images to the GAN latent space and manipulating the latent vector. However, these approaches are limited in that only images from a narrow domain can be transformed, and with only a limited number of editing operations. We propose FlexIT, a novel method which can take any input image and a user-defined text instruction for editing. Our method achieves flexible and natural editing, pushing the limits of semantic image translation. First, FlexIT combines the input image and text into a single target point in the CLIP multimodal embedding space. Via the latent space of an auto-encoder, we iteratively transform the input image toward the target point, ensuring coherence and quality with a variety of novel regularization terms. We propose an evaluation protocol for semantic image translation, and thoroughly evaluate our method on ImageNet. Code will be made publicly available.