# SESAME: Semantic Editing of Scenes by Adding, Manipulating or Erasing Objects

arXiv: https://arxiv.org/abs/2004.04977 · HTML: https://mlsanity.com/paper/2004.04977/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2020.

## Built on by (most cited first, 30 of 43)

- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Enhancing Photorealism Enhancement](https://mlsanity.com/paper/2105.04619/) (2021)
- [CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training](https://mlsanity.com/paper/2203.11947/) (2022)
- [SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis](https://mlsanity.com/paper/2304.14573/) (2023)
- [Bi-level Feature Alignment for Versatile Image Translation and Manipulation](https://mlsanity.com/paper/2107.03021/) (2021)
- [Arbitrary-Scale Image Synthesis](https://mlsanity.com/paper/2204.02273/) (2022)
- [PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis](https://mlsanity.com/paper/2403.01852/) (2024)
- [Deep Learning Technique for Human Parsing: A Survey and Outlook](https://mlsanity.com/paper/2301.00394/) (2023)
- [Semantic-shape Adaptive Feature Modulation for Semantic Image Synthesis](https://mlsanity.com/paper/2203.16898/) (2022)
- [Instruct2Attack: Language-Guided Semantic Adversarial Attacks](https://mlsanity.com/paper/2311.15551/) (2023)
- [Retrieval-based Spatially Adaptive Normalization for Semantic Image Synthesis](https://mlsanity.com/paper/2204.02854/) (2022)
- [EGIC: Enhanced Low-Bit-Rate Generative Image Compression Guided by Semantic Segmentation](https://mlsanity.com/paper/2309.03244/) (2023)
- [Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesis](https://mlsanity.com/paper/2412.20651/) (2024)
- [SIEDOB: Semantic Image Editing by Disentangling Object and Background](https://mlsanity.com/paper/2303.13062/) (2023)
- [Stochastic Conditional Diffusion Models for Robust Semantic Image Synthesis](https://mlsanity.com/paper/2402.16506/) (2024)
- [Context-Consistent Semantic Image Editing with Style-Preserved Modulation](https://mlsanity.com/paper/2207.06252/) (2022)
- [LightsOut: Diffusion-based Outpainting for Enhanced Lens Flare Removal](https://mlsanity.com/paper/2510.15868/) (2025)
- [Layout-to-Image Translation with Double Pooling Generative Adversarial Networks](https://mlsanity.com/paper/2108.12900/) (2021)
- [Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesis](https://mlsanity.com/paper/2307.12084/) (2023)
- [SemIE: Semantically-aware Image Extrapolation](https://mlsanity.com/paper/2108.13702/) (2021)
- [SCP-Diff: Spatial-Categorical Joint Prior for Diffusion Based Semantic Image Synthesis](https://mlsanity.com/paper/2403.09638/) (2024)
- [SAC-GAN: Structure-Aware Image Composition](https://mlsanity.com/paper/2112.06596/) (2021)
- [Semi-supervised reference-based sketch extraction using a contrastive learning framework](https://mlsanity.com/paper/2407.14026/) (2024)
- [Semantic Image Synthesis with Semantically Coupled VQ-Model](https://mlsanity.com/paper/2209.02536/) (2022)
- [DisPositioNet: Disentangled Pose and Identity in Semantic Image Manipulation](https://mlsanity.com/paper/2211.05499/) (2022)
- [ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions](https://mlsanity.com/paper/2205.12231/) (2022)
- [PRISM: Progressive Restoration for Scene Graph-based Image Manipulation](https://mlsanity.com/paper/2311.02247/) (2023)
- [URSimulator: Human-Perception-Driven Prompt Tuning for Enhanced Virtual Urban Renewal via Diffusion Models](https://mlsanity.com/paper/2409.14589/) (2024)
- [Are conditional GANs explicitly conditional?](https://mlsanity.com/paper/2106.15011/) (2021)

## Abstract (arXiv)

Recent advances in image generation gave rise to powerful tools for semantic image editing. However, existing approaches can either operate on a single image or require an abundance of additional information. They are not capable of handling the complete set of editing operations, that is addition, manipulation or removal of semantic concepts. To address these limitations, we propose SESAME, a novel generator-discriminator pair for Semantic Editing of Scenes by Adding, Manipulating or Erasing objects. In our setup, the user provides the semantic labels of the areas to be edited and the generator synthesizes the corresponding pixels. In contrast to previous methods that employ a discriminator that trivially concatenates semantics and image as an input, the SESAME discriminator is composed of two input streams that independently process the image and its semantics, using the latter to manipulate the results of the former. We evaluate our model on a diverse set of datasets and report state-of-the-art performance on two tasks: (a) image manipulation and (b) image generation conditioned on semantic labels.