# FlexEdit: Flexible and Controllable Diffusion-based Object-centric Image Editing

arXiv: https://arxiv.org/abs/2403.18605 · HTML: https://mlsanity.com/paper/2403.18605/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [Text2LIVE: Text-Driven Layered Image and Video Editing](https://mlsanity.com/paper/2204.02491/) (2022)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)
- [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023)
- [EDICT: Exact Diffusion Inversion via Coupled Transformations](https://mlsanity.com/paper/2211.12446/) (2022)
- [InstructDiffusion: A Generalist Modeling Interface for Vision Tasks](https://mlsanity.com/paper/2309.03895/) (2023)
- [SINE: SINgle Image Editing with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2212.04489/) (2022)
- [Learning Rich Features for Image Manipulation Detection](https://mlsanity.com/paper/1805.04953/) (2018)
- [Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code](https://mlsanity.com/paper/2310.01506/) (2023)
- [Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation](https://mlsanity.com/paper/2309.14303/) (2023)
- [SPAN: Spatial Pyramid Attention Network forImage Manipulation Localization](https://mlsanity.com/paper/2009.00726/) (2020)
- [Divide & Bind Your Attention for Improved Generative Semantic Nursing](https://mlsanity.com/paper/2307.10864/) (2023)
- [A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesis](https://mlsanity.com/paper/2306.14544/) (2023)
- [Watch Your Steps: Local Image and Scene Editing by Text Instructions](https://mlsanity.com/paper/2308.08947/) (2023)

## Built on by (most cited first, 12 of 12)

- [GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editing](https://mlsanity.com/paper/2503.10639/) (2025)
- [BrushEdit: All-In-One Image Inpainting and Editing](https://mlsanity.com/paper/2412.10316/) (2024)
- [SwiftEdit: Lightning Fast Text-Guided Image Editing via One-Step Diffusion](https://mlsanity.com/paper/2412.04301/) (2024)
- [Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoning](https://mlsanity.com/paper/2507.01908/) (2025)
- [Discrete Noise Inversion for Next-scale Autoregressive Text-based Image Editing](https://mlsanity.com/paper/2509.01984/) (2025)
- [AttenCraft: Attention-guided Disentanglement of Multiple Concepts for Text-to-Image Customization](https://mlsanity.com/paper/2405.17965/) (2024)
- [A Diffusion-Based Framework for Occluded Object Movement](https://mlsanity.com/paper/2504.01873/) (2025)
- [SmartFreeEdit: Mask-Free Spatial-Aware Image Editing with Complex Instruction Understanding](https://mlsanity.com/paper/2504.12704/) (2025)
- [EditScout: Locating Forged Regions from Diffusion-based Edited Images with Multimodal LLM](https://mlsanity.com/paper/2412.03809/) (2024)
- [InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting](https://mlsanity.com/paper/2603.23463/) (2026)
- [Cross-Space Distillation: Teaching One-Step Students with Modern Diffusion Teachers](https://mlsanity.com/paper/2606.32020/) (2026)
- [SwiftPie: Lightning-fast Subject-driven Image Personalization via One step Diffusion](https://mlsanity.com/paper/2605.01510/) (2026)

## Abstract (arXiv)

Our work addresses limitations seen in previous approaches for object-centric editing problems, such as unrealistic results due to shape discrepancies and limited control in object replacement or insertion. To this end, we introduce FlexEdit, a flexible and controllable editing framework for objects where we iteratively adjust latents at each denoising step using our FlexEdit block. Initially, we optimize latents at test time to align with specified object constraints. Then, our framework employs an adaptive mask, automatically extracted during denoising, to protect the background while seamlessly blending new content into the target image. We demonstrate the versatility of FlexEdit in various object editing tasks and curate an evaluation test suite with samples from both real and synthetic images, along with novel evaluation metrics designed for object-centric editing. We conduct extensive experiments on different editing scenarios, demonstrating the superiority of our editing framework over recent advanced text-guided image editing methods. Our project page is published at https://flex-edit.github.io/.