# Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches

arXiv: https://arxiv.org/abs/2001.02890 · HTML: https://mlsanity.com/paper/2001.02890/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2020.

## Built on by (most cited first, 30 of 37)

- [SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches](https://mlsanity.com/paper/2111.15078/) (2021)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [MagicQuill: An Intelligent Interactive Image Editing System](https://mlsanity.com/paper/2411.09703/) (2024)
- [Learning to Generate Realistic Noisy Images via Pixel-level Noise-aware Adversarial Training](https://mlsanity.com/paper/2204.02844/) (2022)
- [Adversarial Open Domain Adaptation for Sketch-to-Photo Synthesis](https://mlsanity.com/paper/2104.05703/) (2021)
- [RFormer: Transformer-based Generative Adversarial Network for Real Fundus Image Restoration on A New Clinical Benchmark](https://mlsanity.com/paper/2201.00466/) (2022)
- [ScribbleLight: Single Image Indoor Relighting with Scribbles](https://mlsanity.com/paper/2411.17696/) (2024)
- [Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches](https://mlsanity.com/paper/2203.14843/) (2022)
- [Towards Generic Image Manipulation Detection with Weakly-Supervised Self-Consistency Learning](https://mlsanity.com/paper/2309.01246/) (2023)
- [What Can Human Sketches Do for Object Detection?](https://mlsanity.com/paper/2303.15149/) (2023)
- [DrawingInStyles: Portrait Image Generation and Editing with Spatially Conditioned StyleGAN](https://mlsanity.com/paper/2203.02762/) (2022)
- [Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings](https://mlsanity.com/paper/2303.11502/) (2023)
- [Style Your Hair: Latent Optimization for Pose-Invariant Hairstyle Transfer via Local-Style-Aware Hair Alignment](https://mlsanity.com/paper/2208.07765/) (2022)
- [Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera](https://mlsanity.com/paper/2304.06019/) (2023)
- [Barbershop: GAN-based Image Compositing using Segmentation Masks](https://mlsanity.com/paper/2106.01505/) (2021)
- [Facial-Sketch Synthesis: A New Challenge](https://mlsanity.com/paper/2112.15439/) (2021)
- [You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval](https://mlsanity.com/paper/2403.07222/) (2024)
- [How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?](https://mlsanity.com/paper/2403.07203/) (2024)
- [HairFastGAN: Realistic and Robust Hair Transfer with a Fast Encoder-Based Approach](https://mlsanity.com/paper/2404.01094/) (2024)
- [HairFIT: Pose-Invariant Hairstyle Transfer via Flow-based Hair Alignment and Semantic-Region-Aware Inpainting](https://mlsanity.com/paper/2206.08585/) (2022)
- [dualFace:Two-Stage Drawing Guidance for Freehand Portrait Sketching](https://mlsanity.com/paper/2104.12297/) (2021)
- [Shape-guided Object Inpainting](https://mlsanity.com/paper/2204.07845/) (2022)
- [SketchFFusion: Sketch-guided image editing with diffusion model](https://mlsanity.com/paper/2304.03174/) (2023)
- [Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers](https://mlsanity.com/paper/2403.07214/) (2024)
- [AniFaceDrawing: Anime Portrait Exploration during Your Sketching](https://mlsanity.com/paper/2306.07476/) (2023)
- [Towards Interactive Image Inpainting via Sketch Refinement](https://mlsanity.com/paper/2306.00407/) (2023)
- [Black-Box Diagnosis and Calibration on GAN Intra-Mode Collapse: A Pilot Study](https://mlsanity.com/paper/2107.12202/) (2021)
- [ShowFace: Coordinated Face Inpainting with Memory-Disentangled Refinement Networks](https://mlsanity.com/paper/2204.02824/) (2022)
- [Flexible Portrait Image Editing with Fine-Grained Control](https://mlsanity.com/paper/2204.01318/) (2022)
- [Towards Practicality of Sketch-Based Visual Understanding](https://mlsanity.com/paper/2210.15146/) (2022)

## Abstract (arXiv)

Sketch-based image editing aims to synthesize and modify photos based on the structural information provided by the human-drawn sketches. Since sketches are difficult to collect, previous methods mainly use edge maps instead of sketches to train models (referred to as edge-based models). However, sketches display great structural discrepancy with edge maps, thus failing edge-based models. Moreover, sketches often demonstrate huge variety among different users, demanding even higher generalizability and robustness for the editing model to work. In this paper, we propose Deep Plastic Surgery, a novel, robust and controllable image editing framework that allows users to interactively edit images using hand-drawn sketch inputs. We present a sketch refinement strategy, as inspired by the coarse-to-fine drawing process of the artists, which we show can help our model well adapt to casual and varied sketches without the need for real sketch training data. Our model further provides a refinement level control parameter that enables users to flexibly define how "reliable" the input sketch should be considered for the final output, balancing between sketch faithfulness and output verisimilitude (as the two goals might contradict if the input sketch is drawn poorly). To achieve the multi-level refinement, we introduce a style-based module for level conditioning, which allows adaptive feature representations for different levels in a singe network. Extensive experimental results demonstrate the superiority of our approach in improving the visual quality and user controllablity of image editing over the state-of-the-art methods.