# DeFLOCNet: Deep Image Editing via Flexible Low-level Controls

arXiv: https://arxiv.org/abs/2103.12723 · HTML: https://mlsanity.com/paper/2103.12723/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 23 Mar 2021.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [The relativistic discriminator: a key element missing from standard GAN](https://mlsanity.com/paper/1807.00734/) (2018)
- [Semantic Photo Manipulation with a Generative Image Prior](https://mlsanity.com/paper/2005.07727/) (2020)
- [Invertible Conditional GANs for image editing](https://mlsanity.com/paper/1611.06355/) (2016)
- [SC-FEGAN: Face Editing Generative Adversarial Network with User's Sketch and Color](https://mlsanity.com/paper/1902.06838/) (2019)
- [Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations](https://mlsanity.com/paper/2007.06929/) (2020)
- [Coherent Semantic Attention for Image Inpainting](https://mlsanity.com/paper/1905.12384/) (2019)
- [Bringing Old Photos Back to Life](https://mlsanity.com/paper/2004.09484/) (2020)
- [Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language](https://mlsanity.com/paper/1810.11919/) (2018)
- [Shift-Net: Image Inpainting via Deep Feature Rearrangement](https://mlsanity.com/paper/1801.09392/) (2018)
- [Generative Face Completion](https://mlsanity.com/paper/1704.05838/) (2017)
- [VITAL: VIsual Tracking via Adversarial Learning](https://mlsanity.com/paper/1804.04273/) (2018)
- [Mask-Guided Portrait Editing with Conditional GANs](https://mlsanity.com/paper/1905.10346/) (2019)
- [Language-Based Image Editing with Recurrent Attentive Models](https://mlsanity.com/paper/1711.06288/) (2017)
- [CREST: Convolutional Residual Learning for Visual Tracking](https://mlsanity.com/paper/1708.00225/) (2017)
- [Learning Hierarchical Semantic Image Manipulation through Structured Representations](https://mlsanity.com/paper/1808.07535/) (2018)
- [Rethinking Image Deraining via Rain Streaks and Vapors](https://mlsanity.com/paper/2008.00823/) (2020)
- [Joint Face Hallucination and Deblurring via Structure Generation and Detail Enhancement](https://mlsanity.com/paper/1811.09019/) (2018)

## Built on by (most cited first, 26 of 26)

- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021)
- [Follow-Your-Creation: Empowering 4D Creation through Video Inpainting](https://mlsanity.com/paper/2506.04590/) (2025)
- [SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches](https://mlsanity.com/paper/2111.15078/) (2021)
- [Few-shot Image Generation via Adaptation-Aware Kernel Modulation](https://mlsanity.com/paper/2210.16559/) (2022)
- [One Model to Edit Them All: Free-Form Text-Driven Image Manipulation with Semantic Modulations](https://mlsanity.com/paper/2210.07883/) (2022)
- [Delving StyleGAN Inversion for Image Editing: A Foundation Latent Space Viewpoint](https://mlsanity.com/paper/2211.11448/) (2022)
- [Human MotionFormer: Transferring Human Motions with Vision Transformers](https://mlsanity.com/paper/2302.11306/) (2023)
- [EdiBERT, a generative model for image editing](https://mlsanity.com/paper/2111.15264/) (2021)
- [SketchINR: A First Look into Sketches as Implicit Neural Representations](https://mlsanity.com/paper/2403.09344/) (2024)
- [SIEDOB: Semantic Image Editing by Disentangling Object and Background](https://mlsanity.com/paper/2303.13062/) (2023)
- [Context-Consistent Semantic Image Editing with Style-Preserved Modulation](https://mlsanity.com/paper/2207.06252/) (2022)
- [Shape-guided Object Inpainting](https://mlsanity.com/paper/2204.07845/) (2022)
- [SketchFFusion: Sketch-guided image editing with diffusion model](https://mlsanity.com/paper/2304.03174/) (2023)
- [Sketch-guided Image Inpainting with Partial Discrete Diffusion Process](https://mlsanity.com/paper/2404.11949/) (2024)
- [LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model](https://mlsanity.com/paper/2412.11519/) (2024)
- [Point Prompting: Counterfactual Tracking with Video Diffusion Models](https://mlsanity.com/paper/2510.11715/) (2025)
- [Towards Interactive Image Inpainting via Sketch Refinement](https://mlsanity.com/paper/2306.00407/) (2023)
- [What Sketch Explainability Really Means for Downstream Tasks](https://mlsanity.com/paper/2403.09480/) (2024)
- [ShowFace: Coordinated Face Inpainting with Memory-Disentangled Refinement Networks](https://mlsanity.com/paper/2204.02824/) (2022)
- [ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions](https://mlsanity.com/paper/2205.12231/) (2022)
- [Do Generalised Classifiers really work on Human Drawn Sketches?](https://mlsanity.com/paper/2407.03893/) (2024)
- [AdAM: Few-Shot Image Generation via Adaptation-Aware Kernel Modulation](https://mlsanity.com/paper/2307.01465/) (2023)
- [Interactive Neural Painting](https://mlsanity.com/paper/2307.16441/) (2023)
- [Collaborative Neural Painting](https://mlsanity.com/paper/2312.01800/) (2023)
- [Interactive Tumor Progression Modeling via Sketch-Based Image Editing](https://mlsanity.com/paper/2503.06809/) (2025)

## Abstract (arXiv)

User-intended visual content fills the hole regions of an input image in the image editing scenario. The coarse low-level inputs, which typically consist of sparse sketch lines and color dots, convey user intentions for content creation (ie, free-form editing). While existing methods combine an input image and these low-level controls for CNN inputs, the corresponding feature representations are not sufficient to convey user intentions, leading to unfaithfully generated content. In this paper, we propose DeFLOCNet which relies on a deep encoder-decoder CNN to retain the guidance of these controls in the deep feature representations. In each skip-connection layer, we design a structure generation block. Instead of attaching low-level controls to an input image, we inject these controls directly into each structure generation block for sketch line refinement and color propagation in the CNN feature space. We then concatenate the modulated features with the original decoder features for structure generation. Meanwhile, DeFLOCNet involves another decoder branch for texture generation and detail enhancement. Both structures and textures are rendered in the decoder, leading to user-intended editing results. Experiments on benchmarks demonstrate that DeFLOCNet effectively transforms different user intentions to create visually pleasing content.