# SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches

arXiv: https://arxiv.org/abs/2111.15078 · HTML: https://mlsanity.com/paper/2111.15078/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 67 references that are in the map.
- First posted on arXiv 30 Nov 2021.

## Builds on (most cited first, 30 of 49)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Progressive Image Deraining Networks: A Better and Simpler Baseline](https://mlsanity.com/paper/1901.09221/) (2019)
- [Are GANs Created Equal? A Large-Scale Study](https://mlsanity.com/paper/1711.10337/) (2017)
- [Residual Non-local Attention Networks for Image Restoration](https://mlsanity.com/paper/1903.10082/) (2019)
- [Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining](https://mlsanity.com/paper/1807.05698/) (2018)
- [Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset](https://mlsanity.com/paper/1904.01538/) (2019)
- [Pixel Difference Networks for Efficient Edge Detection](https://mlsanity.com/paper/2108.07009/) (2021)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting](https://mlsanity.com/paper/2005.09704/) (2020)
- [StructureFlow: Image Inpainting via Structure-aware Appearance Flow](https://mlsanity.com/paper/1908.03852/) (2019)
- [Cross-domain Correspondence Learning for Exemplar-based Image Translation](https://mlsanity.com/paper/2004.05571/) (2020)
- [SC-FEGAN: Face Editing Generative Adversarial Network with User's Sketch and Color](https://mlsanity.com/paper/1902.06838/) (2019)
- [Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations](https://mlsanity.com/paper/2007.06929/) (2020)
- [High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis](https://mlsanity.com/paper/1611.09969/) (2016)
- [Foreground-aware Image Inpainting](https://mlsanity.com/paper/1901.05945/) (2019)
- [Learning to predict crisp boundaries](https://mlsanity.com/paper/1807.10097/) (2018)
- [SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis](https://mlsanity.com/paper/1801.02753/) (2018)
- [Image Inpainting via Generative Multi-column Convolutional Neural Networks](https://mlsanity.com/paper/1810.08771/) (2018)
- [PD-GAN: Probabilistic Diverse GAN for Image Inpainting](https://mlsanity.com/paper/2105.02201/) (2021)

## Built on by (most cited first, 30 of 33)

- [TALL: Thumbnail Layout for Deepfake Video Detection](https://mlsanity.com/paper/2307.07494/) (2023)
- [Class-Aware Adversarial Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2201.10737/) (2022)
- [MagicQuill: An Intelligent Interactive Image Editing System](https://mlsanity.com/paper/2411.09703/) (2024)
- [ScribbleLight: Single Image Indoor Relighting with Scribbles](https://mlsanity.com/paper/2411.17696/) (2024)
- [Towards Generic Image Manipulation Detection with Weakly-Supervised Self-Consistency Learning](https://mlsanity.com/paper/2309.01246/) (2023)
- [3Doodle: Compact Abstraction of Objects with 3D Strokes](https://mlsanity.com/paper/2402.03690/) (2024)
- [Zooming In on Fakes: A Novel Dataset for Localized AI-Generated Image Detection with Forgery Amplification Approach](https://mlsanity.com/paper/2504.11922/) (2025)
- [How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?](https://mlsanity.com/paper/2403.07203/) (2024)
- [SketchFFusion: Sketch-guided image editing with diffusion model](https://mlsanity.com/paper/2304.03174/) (2023)
- [Sketch-guided Image Inpainting with Partial Discrete Diffusion Process](https://mlsanity.com/paper/2404.11949/) (2024)
- [Repositioning the Subject within Image](https://mlsanity.com/paper/2401.16861/) (2024)
- [Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation](https://mlsanity.com/paper/2305.11317/) (2023)
- [Towards Interactive Image Inpainting via Sketch Refinement](https://mlsanity.com/paper/2306.00407/) (2023)
- [What Sketch Explainability Really Means for Downstream Tasks](https://mlsanity.com/paper/2403.09480/) (2024)
- [Neural-Driven Image Editing](https://mlsanity.com/paper/2507.05397/) (2025)
- [CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing](https://mlsanity.com/paper/2402.17624/) (2024)
- [Fighting Malicious Media Data: A Survey on Tampering Detection and Deepfake Detection](https://mlsanity.com/paper/2212.05667/) (2022)
- [Structure-Guided Image Completion with Image-level and Object-level Semantic Discriminators](https://mlsanity.com/paper/2212.06310/) (2022)
- [Do Generalised Classifiers really work on Human Drawn Sketches?](https://mlsanity.com/paper/2407.03893/) (2024)
- [Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing](https://mlsanity.com/paper/2508.13797/) (2025)
- [A Structure-Guided Diffusion Model for Large-Hole Image Completion](https://mlsanity.com/paper/2211.10437/) (2022)
- [Sketch-based Video Object Segmentation: Benchmark and Analysis](https://mlsanity.com/paper/2311.07261/) (2023)
- [HyperEditor: Achieving Both Authenticity and Cross-Domain Capability in Image Editing via Hypernetworks](https://mlsanity.com/paper/2312.13537/) (2023)
- [SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches](https://mlsanity.com/paper/2501.16022/) (2025)
- [Interactive Tumor Progression Modeling via Sketch-Based Image Editing](https://mlsanity.com/paper/2503.06809/) (2025)
- [Learning Universal Features for Generalizable Image Forgery Localization](https://mlsanity.com/paper/2504.07462/) (2025)
- [FakeRadar: Probing Forgery Outliers to Detect Unknown Deepfake Videos](https://mlsanity.com/paper/2512.14601/) (2025)
- [Position: Natural Language Should Not Fully Replace Formal Languages](https://mlsanity.com/paper/2607.20432/) (2026)
- [Escaping Data Scarcity for High-Resolution Heterogeneous Face Hallucination](https://mlsanity.com/paper/2203.16669/) (2022)
- [SketchAssist: A Practical Assistant for Semantic Edits and Precise Local Redrawing](https://mlsanity.com/paper/2512.14140/) (2025)

## Abstract (arXiv)

Sketch-based image manipulation is an interactive image editing task to modify an image based on input sketches from users. Existing methods typically formulate this task as a conditional inpainting problem, which requires users to draw an extra mask indicating the region to modify in addition to sketches. The masked regions are regarded as holes and filled by an inpainting model conditioned on the sketch. With this formulation, paired training data can be easily obtained by randomly creating masks and extracting edges or contours. Although this setup simplifies data preparation and model design, it complicates user interaction and discards useful information in masked regions. To this end, we investigate a new paradigm of sketch-based image manipulation: mask-free local image manipulation, which only requires sketch inputs from users and utilizes the entire original image. Given an image and sketch, our model automatically predicts the target modification region and encodes it into a structure agnostic style vector. A generator then synthesizes the new image content based on the style vector and sketch. The manipulated image is finally produced by blending the generator output into the modification region of the original image. Our model can be trained in a self-supervised fashion by learning the reconstruction of an image region from the style vector and sketch. The proposed method offers simpler and more intuitive user workflows for sketch-based image manipulation and provides better results than previous approaches. More results, code and interactive demo will be available at urlhttps://zengxianyu.github.io/sketchedit.