# SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editing

arXiv: https://arxiv.org/abs/2410.11815 · HTML: https://mlsanity.com/paper/2410.11815/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 56 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 30 of 31)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)
- [GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions](https://mlsanity.com/paper/2104.14806/) (2021)
- [Break-A-Scene: Extracting Multiple Concepts from a Single Image](https://mlsanity.com/paper/2305.16311/) (2023)
- [Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs](https://mlsanity.com/paper/2401.11708/) (2024)
- [Dense Text-to-Image Generation with Attention Modulation](https://mlsanity.com/paper/2308.12964/) (2023)
- [LayoutTransformer: Layout Generation and Completion with Self-attention](https://mlsanity.com/paper/2006.14615/) (2020)
- [Intriguing properties of synthetic images: from generative adversarial networks to diffusion models](https://mlsanity.com/paper/2304.06408/) (2023)
- [LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation](https://mlsanity.com/paper/2308.05095/) (2023)
- [Semantic Image Manipulation Using Scene Graphs](https://mlsanity.com/paper/2004.03677/) (2020)
- [EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods](https://mlsanity.com/paper/2310.02426/) (2023)
- [SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis](https://mlsanity.com/paper/2304.14573/) (2023)
- [Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generation](https://mlsanity.com/paper/2310.08541/) (2023)
- [FlowZero: Zero-Shot Text-to-Video Synthesis with LLM-Driven Dynamic Scene Syntax](https://mlsanity.com/paper/2311.15813/) (2023)
- [De-Diffusion Makes Text a Strong Cross-Modal Interface](https://mlsanity.com/paper/2311.00618/) (2023)
- [LECA: A Learned Approach for Efficient Cover-agnostic Watermarking](https://mlsanity.com/paper/2206.10813/) (2022)

## Built on by (most cited first, 10 of 10)

- [A Unified Agentic Framework for Evaluating Conditional Image Generation](https://mlsanity.com/paper/2504.07046/) (2025)
- [MoEdit: On Learning Quantity Perception for Multi-object Image Editing](https://mlsanity.com/paper/2503.10112/) (2025)
- [CoSTAast: Cost-Sensitive Toolpath Agent for Multi-turn Image Editing](https://mlsanity.com/paper/2503.10613/) (2025)
- [Designing Memory-Augmented AR Agents for Spatiotemporal Reasoning in Personalized Task Assistance](https://mlsanity.com/paper/2508.08774/) (2025)
- [Range-Edit: Semantic Mask Guided Outdoor LiDAR Scene Editing](https://mlsanity.com/paper/2511.17269/) (2025)
- [Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrieval](https://mlsanity.com/paper/2604.15951/) (2026)
- [SceneCraft: Interactive System for Image Editing via Scene Graph](https://mlsanity.com/paper/2606.16103/) (2026)
- [SAGE: Scene Graph-Aware Guidance and Execution for Long-Horizon Manipulation Tasks](https://mlsanity.com/paper/2509.21928/) (2025)
- [VENUS: Visual Editing with Noise Inversion Using Scene Graphs](https://mlsanity.com/paper/2601.07219/) (2026)
- [SimGraph: A Unified Framework for Scene Graph-Based Image Generation and Editing](https://mlsanity.com/paper/2601.21498/) (2026)

## Abstract (arXiv)

Scene graphs offer a structured, hierarchical representation of images, with nodes and edges symbolizing objects and the relationships among them. It can serve as a natural interface for image editing, dramatically improving precision and flexibility. Leveraging this benefit, we introduce a new framework that integrates large language model (LLM) with Text2Image generative model for scene graph-based image editing. This integration enables precise modifications at the object level and creative recomposition of scenes without compromising overall image integrity. Our approach involves two primary stages: 1) Utilizing a LLM-driven scene parser, we construct an image's scene graph, capturing key objects and their interrelationships, as well as parsing fine-grained attributes such as object masks and descriptions. These annotations facilitate concept learning with a fine-tuned diffusion model, representing each object with an optimized token and detailed description prompt. 2) During the image editing phase, a LLM editing controller guides the edits towards specific areas. These edits are then implemented by an attention-modulated diffusion editor, utilizing the fine-tuned model to perform object additions, deletions, replacements, and adjustments. Through extensive experiments, we demonstrate that our framework significantly outperforms existing image editing methods in terms of editing precision and scene aesthetics.