mlsanityData to 7 Oct 2026

Reference-based Image Composition with Sketch via Structure-aware Diffusion Model

arXiv 2304.09748 ↗First posted 31 Mar 2023cs.CV cs.AIcited by 16

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent remarkable improvements in large-scale text-to-image generative models have shown promising results in generating high-fidelity images. To further enhance editability and enable fine-grained generation, we introduce a multi-input-conditioned image composition model that incorporates a sketch as a novel modal, alongside a reference image. Thanks to the edge-level controllability using sketches, our method enables a user to edit or complete an image sub-part with a desired structure (i.e., sketch) and content (i.e., reference image). Our framework fine-tunes a pre-trained diffusion model to complete missing regions using the reference image while maintaining sketch guidance. Albeit simple, this leads to wide opportunities to fulfill user needs for obtaining the in-demand images. Through extensive experiments, we demonstrate that our proposed method offers unique use cases for image manipulation, enabling user-driven modifications of arbitrary scenes.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 13 of its 17 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

13 of 17 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  3. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  4. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Models23 Nov
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 6

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  5. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  6. 2021Pixel Difference Networks for Efficient Edge Detectioncited by 167

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 8; 2025: 6.

16 papers.

Sort:

By area:

  1. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  2. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
  3. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  4. 2024Thinking Outside the BBox: Unconstrained Generative Object Compositingcited by 19
  5. 2023Integrating View Conditions for Image Synthesiscited by 16
  6. 2024What to Preserve and What to Transfer: Faithful, Identity-Preserving Diffusion-based Hairstyle Transfercited by 7
  7. 2023DIFF-NST: Diffusion Interleaving For deFormable Neural Style Transfercited by 5
  8. 2025Image Referenced Sketch Colorization Based on Animation Creation Workflowcited by 5
  9. 2025Free-Form Scene Editor: Enabling Multi-Round Object Manipulation like in a 3D Enginecited by 3
  10. 2024Sketch2Human: Deep Human Generation with Disentangled Geometry and Appearance Controlcited by 1
  11. 2024DiffInject: Revisiting Debias via Synthetic Data Generation using Diffusion-based Style Injectioncited by 1
  12. 20252D Instance Editing in 3D Spacecited by 1
4 more
  1. 2024ControlEdit: A MultiModal Local Clothing Image Editing Methodcited by 0
  2. 2025Recovering Partially Corrupted Objects via Sketch-Guided Bidirectional Feature Interactioncited by 0
  3. 2025ColorizeDiffusion v2: Enhancing Reference-based Sketch Colorization Through Separating Utilitiescited by 0
  4. 2025ReMix: Towards a Unified View of Consistent Character Generation and Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023SketchFFusion: Sketch-guided image editing with diffusion model
  2. 2023Text-Guided Scene Sketch-to-Photo Synthesis
  3. 2024Training-Free Sketch-Guided Diffusion with Latent Optimization
  4. 2024Sketch-Guided Scene Image Generation
  5. 2024U-Sketch: An Efficient Approach for Sketch to Image Diffusion Models
  6. 2024Generating Compositional Scenes via Text-to-image RGBA Instance Generation
  7. 2024Draw Like an Artist: Complex Scene Generation with Diffusion Model via Composition, Painting, and Retouching
  8. 2024KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Models