# SketchFFusion: Sketch-guided image editing with diffusion model

arXiv: https://arxiv.org/abs/2304.03174 · HTML: https://mlsanity.com/paper/2304.03174/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 6 Apr 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 18 of 18)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [SC-FEGAN: Face Editing Generative Adversarial Network with User's Sketch and Color](https://mlsanity.com/paper/1902.06838/) (2019)
- [Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations](https://mlsanity.com/paper/2007.06929/) (2020)
- [Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting](https://mlsanity.com/paper/1904.07475/) (2019)
- [Image inpainting: A review](https://mlsanity.com/paper/1909.06399/) (2019)
- [Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches](https://mlsanity.com/paper/2001.02890/) (2020)
- [SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches](https://mlsanity.com/paper/2111.15078/) (2021)
- [DeFLOCNet: Deep Image Editing via Flexible Low-level Controls](https://mlsanity.com/paper/2103.12723/) (2021)
- [Smart, Sparse Contours to Represent and Edit Images](https://mlsanity.com/paper/1712.08232/) (2017)
- [Image Fine-grained Inpainting](https://mlsanity.com/paper/2002.02609/) (2020)

## Built on by (most cited first, 8 of 8)

- [MagicQuill: An Intelligent Interactive Image Editing System](https://mlsanity.com/paper/2411.09703/) (2024)
- [A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trends](https://mlsanity.com/paper/2402.15490/) (2024)
- [A Comprehensive Survey on Diffusion Models and Their Applications](https://mlsanity.com/paper/2408.10207/) (2024)
- [StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback](https://mlsanity.com/paper/2510.20093/) (2025)
- [Multi-Accent Mandarin Dry-Vocal Singing Dataset: Benchmark for Singing Accent Recognition](https://mlsanity.com/paper/2512.07005/) (2025)
- [Rethinking Scribble-Guided Image Editing: Generalization, Instruction Adherence, and Multi-Tasking](https://mlsanity.com/paper/2605.25568/) (2026)
- [Singing Timbre Popularity Assessment Based on Multimodal Large Foundation Model](https://mlsanity.com/paper/2512.06999/) (2025)
- [ScribbleEdit: Synthetic Data for Image Editing with Scribbles and Text](https://mlsanity.com/paper/2605.01135/) (2026)

## Abstract (arXiv)

Sketch-guided image editing aims to achieve local fine-tuning of the image based on the sketch information provided by the user, while maintaining the original status of the unedited areas. Due to the high cost of acquiring human sketches, previous works mostly relied on edge maps as a substitute for sketches, but sketches possess more rich structural information. In this paper, we propose a sketch generation scheme that can preserve the main contours of an image and closely adhere to the actual sketch style drawn by the user. Simultaneously, current image editing methods often face challenges such as image distortion, training cost, and loss of fine details in the sketch. To address these limitations, We propose a conditional diffusion model (SketchFFusion) based on the sketch structure vector. We evaluate the generative performance of our model and demonstrate that it outperforms existing methods.