mlsanityData to 7 Oct 2026

Editable Image Elements for Controllable Synthesis

arXiv 2404.16029 ↗First posted 24 Apr 2024cs.CVcited by 13

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models have made significant advances in text-guided synthesis tasks. However, editing user-provided images remains challenging, as the high dimensional noise input space of diffusion models is not naturally suited for image inversion or spatial editing. In this work, we propose an image representation that promotes spatial editing of input images using a diffusion model. Concretely, we learn to encode an input into "image elements" that can faithfully reconstruct an input image. These elements can be intuitively edited by a user, and are decoded by a diffusion model into realistic images. We show the effectiveness of our representation on various image editing tasks, such as object resizing, rearrangement, dragging, de-occlusion, removal, variation, and image composition. Project page: https://jitengmu.github.io/Editable_Image_Elements/

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 45 of its 48 references that are in the map. First posted on arXiv 24 Apr 2024.

Builds on

45 of 48 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models23 Feb
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  3. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  4. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  5. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
  6. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  7. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  8. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Models23 Nov
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  10. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  11. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
3 more
  1. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2023Segment Anythingcited by 8,158
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  12. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
18 more
  1. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  2. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  3. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  4. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  5. 2023Zero-shot Image-to-Image Translationcited by 503
  6. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  7. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  8. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 256
  9. 2020Swapping Autoencoder for Deep Image Manipulationcited by 215
  10. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  11. 2018Image Generation from Layoutcited by 105
  12. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  13. 2020Object-Centric Image Generation from Layoutscited by 57
  14. 2020Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesiscited by 49
  15. 2022Modeling Image Composition for Complex Scene Generationcited by 43
  16. 2022BlobGAN: Spatially Disentangled Scene Representationscited by 35
  17. 2021High-Resolution Complex Scene Synthesis with Transformerscited by 31
  18. 2023BlobGAN-3D: A Spatially-Disentangled 3D-Aware Generative Model for Indoor Scenescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 5; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rolloutcited by 47
  2. 2025EditAR: Unified Conditional Generation with Autoregressive Modelscited by 40
  3. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  4. 2025Training-free Geometric Image Editing on Diffusion Modelscited by 13
  5. 2024AdaptiveDrag: Semantic-Driven Dragging on Diffusion-Based Image Editingcited by 8
  6. 2024LoRA of Change: Learning to Generate LoRA for the Editing Instruction from A Single Before-After Image Paircited by 5
  7. 2025Coupled Diffusion Sampling for Training-Free Multi-View Image Editingcited by 4
  8. 2025Free-Form Scene Editor: Enabling Multi-Round Object Manipulation like in a 3D Enginecited by 3
  9. 2026Referring Layer Decompositioncited by 3
  10. 2026OrthoEraser: Coupled-Neuron Orthogonal Projection for Concept Erasurecited by 3
  11. 2025Generative Blocks World: Moving Things Around in Picturescited by 2
  12. 2024GenHeld: Generating and Editing Handheld Objectscited by 0
1 more
  1. 2024Pattern Analogies: Learning to Perform Programmatic Image Edits by Analogycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Image Editing with Diffusion Models: A Survey
  2. 2024Diffusion Model-Based Image Editing: A Survey
  3. 2024Eta Inversion: Designing an Optimal Eta Function for Diffusion-based Real Image Editing
  4. 2026On the Controllability-Fidelity Frontier in Diffusion Editing
  5. 2023Guided Image Synthesis via Initial Image Editing in Diffusion Model
  6. 2023Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models
  7. 2023Differential Diffusion: Giving Each Pixel Its Strength
  8. 2024FunEditor: Achieving Complex Image Edits via Function Aggregation with Diffusion Models