# Pixelated Semantic Colorization

arXiv: https://arxiv.org/abs/1901.10889 · HTML: https://mlsanity.com/paper/1901.10889/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2019.

## Built on by (most cited first, 28 of 28)

- [Towards Vivid and Diverse Image Colorization with Generative Color Prior](https://mlsanity.com/paper/2108.08826/) (2021)
- [DDColor: Towards Photo-Realistic Image Colorization via Dual Decoders](https://mlsanity.com/paper/2212.11613/) (2022)
- [Control Color: Multimodal Diffusion-based Interactive Image Colorization](https://mlsanity.com/paper/2402.10855/) (2024)
- [Improved Diffusion-based Image Colorization via Piggybacked Models](https://mlsanity.com/paper/2304.11105/) (2023)
- [VCGAN: Video Colorization with Hybrid Generative Adversarial Network](https://mlsanity.com/paper/2104.12357/) (2021)
- [Deep Edge-Aware Interactive Colorization against Color-Bleeding Effects](https://mlsanity.com/paper/2107.01619/) (2021)
- [LiftPool: Bidirectional ConvNet Pooling](https://mlsanity.com/paper/2104.00996/) (2021)
- [Video Colorization with Pre-trained Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.01732/) (2023)
- [Follow-Your-Color: Multi-Instance Sketch Colorization](https://mlsanity.com/paper/2503.16948/) (2025)
- [Wavelet Transform-assisted Adaptive Generative Modeling for Colorization](https://mlsanity.com/paper/2107.04261/) (2021)
- [SCSNet: An Efficient Paradigm for Learning Simultaneously Image Colorization and Super-Resolution](https://mlsanity.com/paper/2201.04364/) (2022)
- [Colorizing Monochromatic Radiance Fields](https://mlsanity.com/paper/2402.12184/) (2024)
- [Automatic Controllable Colorization via Imagination](https://mlsanity.com/paper/2404.05661/) (2024)
- [Focusing on Persons: Colorizing Old Images Learning from Modern Historical Movies](https://mlsanity.com/paper/2108.06515/) (2021)
- [L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors](https://mlsanity.com/paper/2305.15217/) (2023)
- [ChromaDistill: Colorizing Monochrome Radiance Fields with Knowledge Distillation](https://mlsanity.com/paper/2309.07668/) (2023)
- [Wallpaper Texture Generation and Style Transfer Based on Multi-label Semantics](https://mlsanity.com/paper/2106.11482/) (2021)
- [CHIMLE: Conditional Hierarchical IMLE for Multimodal Conditional Image Synthesis](https://mlsanity.com/paper/2211.14286/) (2022)
- [SVCNet: Scribble-based Video Colorization Network with Temporal Aggregation](https://mlsanity.com/paper/2303.11591/) (2023)
- [Brighten-and-Colorize: A Decoupled Network for Customized Low-Light Image Enhancement](https://mlsanity.com/paper/2308.03029/) (2023)
- [An Analysis of Generative Methods for Multiple Image Inpainting](https://mlsanity.com/paper/2205.02146/) (2022)
- [ViT-Inception-GAN for Image Colourising](https://mlsanity.com/paper/2106.06321/) (2021)
- [Towards Photorealistic Colorization by Imagination](https://mlsanity.com/paper/2108.09195/) (2021)
- [Human vs Objective Evaluation of Colourisation Performance](https://mlsanity.com/paper/2204.05200/) (2022)
- [iColoriT: Towards Propagating Local Hint to the Right Region in Interactive Colorization by Leveraging Vision Transformer](https://mlsanity.com/paper/2207.06831/) (2022)
- [Audio-Infused Automatic Image Colorization by Exploiting Audio Scene Semantics](https://mlsanity.com/paper/2401.13270/) (2024)
- [Computer-aided Colorization State-of-the-science: A Survey](https://mlsanity.com/paper/2410.02288/) (2024)
- [Instance-aware Image Colorization with Controllable Textual Descriptions and Segmentation Masks](https://mlsanity.com/paper/2505.08705/) (2025)

## Abstract (arXiv)

While many image colorization algorithms have recently shown the capability of producing plausible color versions from gray-scale photographs, they still suffer from limited semantic understanding. To address this shortcoming, we propose to exploit pixelated object semantics to guide image colorization. The rationale is that human beings perceive and distinguish colors based on the semantic categories of objects. Starting from an autoregressive model, we generate image color distributions, from which diverse colored results are sampled. We propose two ways to incorporate object semantics into the colorization model: through a pixelated semantic embedding and a pixelated semantic generator. Specifically, the proposed convolutional neural network includes two branches. One branch learns what the object is, while the other branch learns the object colors. The network jointly optimizes a color embedding loss, a semantic segmentation loss and a color generation loss, in an end-to-end fashion. Experiments on PASCAL VOC2012 and COCO-stuff reveal that our network, when trained with semantic segmentation labels, produces more realistic and finer results compared to the colorization state-of-the-art.