# Improving Sketch Colorization using Adversarial Segmentation Consistency

arXiv: https://arxiv.org/abs/2301.08590 · HTML: https://mlsanity.com/paper/2301.08590/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 20 Jan 2023.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [DualGAN: Unsupervised Dual Learning for Image-to-Image Translation](https://mlsanity.com/paper/1704.02510/) (2017)
- [SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis](https://mlsanity.com/paper/1801.02753/) (2018)
- [Scribbler: Controlling Deep Image Synthesis with Sketch and Color](https://mlsanity.com/paper/1612.00835/) (2016)
- [SketchyCOCO: Image Generation from Freehand Scene Sketches](https://mlsanity.com/paper/2003.02683/) (2020)
- [Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence](https://mlsanity.com/paper/2005.05207/) (2020)
- [TextureGAN: Controlling Deep Image Synthesis with Texture Patches](https://mlsanity.com/paper/1706.02823/) (2017)
- [Unsupervised Sketch-to-Photo Synthesis](https://mlsanity.com/paper/1909.08313/) (2019)
- [Tag2Pix: Line Art Colorization Using Text Tag With SECat and Changing Loss](https://mlsanity.com/paper/1908.05840/) (2019)
- [Image Generation from Sketch Constraint Using Contextual GAN](https://mlsanity.com/paper/1711.08972/) (2017)
- [SketchyScene: Richly-Annotated Scene Sketches](https://mlsanity.com/paper/1808.02473/) (2018)
- [User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks](https://mlsanity.com/paper/1808.03240/) (2018)
- [Deep Generation of Face Images from Sketches](https://mlsanity.com/paper/2006.01047/) (2020)
- [Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN](https://mlsanity.com/paper/1706.03319/) (2017)
- [LinesToFacePhoto: Face Photo Generation from Lines with Conditional Self-Attention Generative Adversarial Network](https://mlsanity.com/paper/1910.08914/) (2019)
- [Comicolorization: Semi-Automatic Manga Colorization](https://mlsanity.com/paper/1706.06759/) (2017)
- [GANILLA: Generative Adversarial Networks for Image to Illustration Translation](https://mlsanity.com/paper/2002.05638/) (2020)
- [Deep Edge-Aware Interactive Colorization against Color-Bleeding Effects](https://mlsanity.com/paper/2107.01619/) (2021)
- [Sketch-to-Art: Synthesizing Stylized Art Images From Sketches](https://mlsanity.com/paper/2002.12888/) (2020)
- [Multi-Density Sketch-to-Image Translation Network](https://mlsanity.com/paper/2006.10649/) (2020)
- [Adversarial Segmentation Loss for Sketch Colorization](https://mlsanity.com/paper/2102.06192/) (2021)

## Built on by (most cited first, 3 of 3)

- [Locally-Focused Face Representation for Sketch-to-Image Generation Using Noise-Induced Refinement](https://mlsanity.com/paper/2411.19005/) (2024)
- [Diverse Feature Learning by Self-distillation and Reset](https://mlsanity.com/paper/2403.19941/) (2024)
- [Component-Aware Sketch-to-Image Generation Using Self-Attention Encoding and Coordinate-Preserving Fusion](https://mlsanity.com/paper/2603.09484/) (2026)

## Abstract (arXiv)

We propose a new method for producing color images from sketches. Current solutions in sketch colorization either necessitate additional user instruction or are restricted to the "paired" translation strategy. We leverage semantic image segmentation from a general-purpose panoptic segmentation network to generate an additional adversarial loss function. The proposed loss function is compatible with any GAN model. Our method is not restricted to datasets with segmentation labels and can be applied to unpaired translation tasks as well. Using qualitative, and quantitative analysis, and based on a user study, we demonstrate the efficacy of our method on four distinct image datasets. On the FID metric, our model improves the baseline by up to 35 points. Our code, pretrained models, scripts to produce newly introduced datasets and corresponding sketch images are available at https://github.com/giddyyupp/AdvSegLoss.