# Semantic Image Synthesis via Adversarial Learning

arXiv: https://arxiv.org/abs/1707.06873 · HTML: https://mlsanity.com/paper/1707.06873/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2017.

## Built on by (most cited first, 30 of 87)

- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Text2LIVE: Text-Driven Layered Image and Video Editing](https://mlsanity.com/paper/2204.02491/) (2022)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [More Control for Free! Image Synthesis with Semantic Diffusion Guidance](https://mlsanity.com/paper/2112.05744/) (2021)
- [EnvEdit: Environment Editing for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.15685/) (2022)
- [Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generation](https://mlsanity.com/paper/2303.09319/) (2023)
- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [StyleMC: Multi-Channel Based Fast Text-Guided Image Generation and Manipulation](https://mlsanity.com/paper/2112.08493/) (2021)
- [Highly Personalized Text Embedding for Image Manipulation by Stable Diffusion](https://mlsanity.com/paper/2303.08767/) (2023)
- [DreamEdit: Subject-driven Image Editing](https://mlsanity.com/paper/2306.12624/) (2023)
- [M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers](https://mlsanity.com/paper/2105.14211/) (2021)
- [Sound-Guided Semantic Image Manipulation](https://mlsanity.com/paper/2112.00007/) (2021)
- [Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model](https://mlsanity.com/paper/2111.13333/) (2021)
- [AnyFace: Free-style Text-to-Face Synthesis and Manipulation](https://mlsanity.com/paper/2203.15334/) (2022)
- [High-Fidelity 3D Face Generation from Natural Language Descriptions](https://mlsanity.com/paper/2305.03302/) (2023)
- [Cycle-Consistent Inverse GAN for Text-to-Image Synthesis](https://mlsanity.com/paper/2108.01361/) (2021)
- [DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulation](https://mlsanity.com/paper/2303.06285/) (2023)
- [Towards Open-World Text-Guided Face Image Generation and Manipulation](https://mlsanity.com/paper/2104.08910/) (2021)
- [Learning by Planning: Language-Guided Global Image Editing](https://mlsanity.com/paper/2106.13156/) (2021)
- [Region-Aware Diffusion for Zero-shot Text-driven Image Editing](https://mlsanity.com/paper/2302.11797/) (2023)
- [Improving Compositional Text-to-image Generation with Large Vision-Language Models](https://mlsanity.com/paper/2310.06311/) (2023)
- [Word-Level Fine-Grained Story Visualization](https://mlsanity.com/paper/2208.02341/) (2022)
- [Learning Visual Styles from Audio-Visual Associations](https://mlsanity.com/paper/2205.05072/) (2022)
- [FICE: Text-Conditioned Fashion Image Editing With Guided GAN Inversion](https://mlsanity.com/paper/2301.02110/) (2023)
- [Towers of Babel: Combining Images, Language, and 3D Geometry for Learning Multimodal Vision](https://mlsanity.com/paper/2108.05863/) (2021)
- [Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation](https://mlsanity.com/paper/2306.08247/) (2023)
- [CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images](https://mlsanity.com/paper/2104.05981/) (2021)
- [PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes](https://mlsanity.com/paper/2106.00446/) (2021)

## Abstract (arXiv)

In this paper, we propose a way of synthesizing realistic images directly with natural language description, which has many useful applications, e.g. intelligent image manipulation. We attempt to accomplish such synthesis: given a source image and a target text description, our model synthesizes images to meet two requirements: 1) being realistic while matching the target text description; 2) maintaining other image features that are irrelevant to the text description. The model should be able to disentangle the semantic information from the two modalities (image and text), and generate new images from the combined semantics. To achieve this, we proposed an end-to-end neural architecture that leverages adversarial learning to automatically learn implicit loss functions, which are optimized to fulfill the aforementioned two requirements. We have evaluated our model by conducting experiments on Caltech-200 bird dataset and Oxford-102 flower dataset, and have demonstrated that our model is capable of synthesizing realistic images that match the given descriptions, while still maintain other features of original images.