# Semantic Image Manipulation Using Scene Graphs

arXiv: https://arxiv.org/abs/2004.03677 · HTML: https://mlsanity.com/paper/2004.03677/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2020.

## Built on by (most cited first, 30 of 82)

- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [More Control for Free! Image Synthesis with Semantic Diffusion Guidance](https://mlsanity.com/paper/2112.05744/) (2021)
- [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021)
- [PSCC-Net: Progressive Spatio-Channel Correlation Network for Image Manipulation Detection and Localization](https://mlsanity.com/paper/2103.10596/) (2021)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [Street-View Image Generation from a Bird's-Eye View Layout](https://mlsanity.com/paper/2301.04634/) (2023)
- [Motion-Conditioned Diffusion Model for Controllable Video Synthesis](https://mlsanity.com/paper/2304.14404/) (2023)
- [Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs](https://mlsanity.com/paper/2108.08841/) (2021)
- [ObjectFormer for Image Manipulation Detection and Localization](https://mlsanity.com/paper/2203.14681/) (2022)
- [Neo-GNNs: Neighborhood Overlap-aware Graph Neural Networks for Link Prediction](https://mlsanity.com/paper/2206.04216/) (2022)
- [Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2203.09811/) (2022)
- [EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusion](https://mlsanity.com/paper/2405.00915/) (2024)
- [Learning to Generate Scene Graph from Natural Language Supervision](https://mlsanity.com/paper/2109.02227/) (2021)
- [Neuro-Symbolic Artificial Intelligence: Current Trends](https://mlsanity.com/paper/2105.05330/) (2021)
- [CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion](https://mlsanity.com/paper/2305.16283/) (2023)
- [4D-OR: Semantic Scene Graphs for OR Domain Modeling](https://mlsanity.com/paper/2203.11937/) (2022)
- [Modeling Image Composition for Complex Scene Generation](https://mlsanity.com/paper/2206.00923/) (2022)
- [SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis](https://mlsanity.com/paper/2304.14573/) (2023)
- [Incremental 3D Semantic Scene Graph Prediction from RGB Sequences](https://mlsanity.com/paper/2305.02743/) (2023)
- [Latent Graph Representations for Critical View of Safety Assessment](https://mlsanity.com/paper/2212.04155/) (2022)
- [FlexIT: Towards Flexible Semantic Image Translation](https://mlsanity.com/paper/2203.04705/) (2022)
- [SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene Graphs](https://mlsanity.com/paper/2309.12188/) (2023)
- [Unconditional Scene Graph Generation](https://mlsanity.com/paper/2108.05884/) (2021)
- [SGRec3D: Self-Supervised 3D Scene Graph Learning via Object-Level Scene Reconstruction](https://mlsanity.com/paper/2309.15702/) (2023)
- [TextPSG: Panoptic Scene Graph Generation from Textual Descriptions](https://mlsanity.com/paper/2310.07056/) (2023)
- [Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning](https://mlsanity.com/paper/2312.06829/) (2023)
- [Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesis](https://mlsanity.com/paper/2412.20651/) (2024)
- [Text-to-image Editing by Image Information Removal](https://mlsanity.com/paper/2305.17489/) (2023)
- [VISAGE: Video Synthesis using Action Graphs for Surgery](https://mlsanity.com/paper/2410.17751/) (2024)

## Abstract (arXiv)

Image manipulation can be considered a special case of image generation where the image to be produced is a modification of an existing image. Image generation and manipulation have been, for the most part, tasks that operate on raw pixels. However, the remarkable progress in learning rich image and object representations has opened the way for tasks such as text-to-image or layout-to-image generation that are mainly driven by semantics. In our work, we address the novel problem of image manipulation from scene graphs, in which a user can edit images by merely applying changes in the nodes or edges of a semantic graph that is generated from the image. Our goal is to encode image information in a given constellation and from there on generate new constellations, such as replacing objects or even changing relationships between objects, while respecting the semantics and style from the original image. We introduce a spatio-semantic scene graph network that does not require direct supervision for constellation changes or image edits. This makes it possible to train the system from existing real-world datasets with no additional annotation effort.