# Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation

arXiv: https://arxiv.org/abs/1804.04128 · HTML: https://mlsanity.com/paper/1804.04128/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2018.

## Built on by (most cited first, 30 of 35)

- [Towards Vivid and Diverse Image Colorization with Generative Color Prior](https://mlsanity.com/paper/2108.08826/) (2021)
- [PalGAN: Image Colorization with Palette Generative Adversarial Networks](https://mlsanity.com/paper/2210.11204/) (2022)
- [Improved Diffusion-based Image Colorization via Piggybacked Models](https://mlsanity.com/paper/2304.11105/) (2023)
- [Semantic-Sparse Colorization Network for Deep Exemplar-based Colorization](https://mlsanity.com/paper/2112.01335/) (2021)
- [Color Recommendation for Vector Graphic Documents based on Multi-Palette Representation](https://mlsanity.com/paper/2209.10820/) (2022)
- [UniColor: A Unified Framework for Multi-Modal Colorization with Transformer](https://mlsanity.com/paper/2209.11223/) (2022)
- [Automatic Controllable Colorization via Imagination](https://mlsanity.com/paper/2404.05661/) (2024)
- [DesignProbe: A Graphic Design Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2404.14801/) (2024)
- [NamedCurves: Learned Image Enhancement via Color Naming](https://mlsanity.com/paper/2407.09892/) (2024)
- [Focusing on Persons: Colorizing Old Images Learning from Modern Historical Movies](https://mlsanity.com/paper/2108.06515/) (2021)
- [Language-based Photo Color Adjustment for Graphic Designs](https://mlsanity.com/paper/2308.03059/) (2023)
- [ColorFlow: Retrieval-Augmented Image Sequence Colorization](https://mlsanity.com/paper/2412.11815/) (2024)
- [Interactive Image Manipulation with Complex Text Instructions](https://mlsanity.com/paper/2211.15352/) (2022)
- [Multimodal Color Recommendation in Vector Graphic Documents](https://mlsanity.com/paper/2308.04118/) (2023)
- [PostDoc: Generating Poster from a Long Multimodal Document Using Deep Submodular Optimization](https://mlsanity.com/paper/2405.20213/) (2024)
- [Generating Compositional Color Representations from Text](https://mlsanity.com/paper/2109.10477/) (2021)
- [SVCNet: Scribble-based Video Colorization Network with Temporal Aggregation](https://mlsanity.com/paper/2303.11591/) (2023)
- [DiffColor: Toward High Fidelity Text-Guided Image Colorization with Diffusion Models](https://mlsanity.com/paper/2308.01655/) (2023)
- [TIC: Text-Guided Image Colorization](https://mlsanity.com/paper/2208.02843/) (2022)
- [From Fragment to One Piece: A Survey on AI-Driven Graphic Design](https://mlsanity.com/paper/2503.18641/) (2025)
- [BBQ-to-Image: Numeric Bounding Box and Qolor Control in Large-Scale Text-to-Image Models](https://mlsanity.com/paper/2602.20672/) (2026)
- [Separating Content and Style for Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2110.14404/) (2021)
- [Flexible Portrait Image Editing with Fine-Grained Control](https://mlsanity.com/paper/2204.01318/) (2022)
- [Name Your Colour For the Task: Artificially Discover Colour Naming via Colour Quantisation Transformer](https://mlsanity.com/paper/2212.03434/) (2022)
- [CCC: Color Classified Colorization](https://mlsanity.com/paper/2403.01476/) (2024)
- [Culture-inspired Multi-modal Color Palette Generation and Colorization: A Chinese Youth Subculture Case](https://mlsanity.com/paper/2102.05231/) (2021)
- [ViT-Inception-GAN for Image Colourising](https://mlsanity.com/paper/2106.06321/) (2021)
- [MMC: Multi-Modal Colorization of Images using Textual Descriptions](https://mlsanity.com/paper/2304.11993/) (2023)
- [Audio-Infused Automatic Image Colorization by Exploiting Audio Scene Semantics](https://mlsanity.com/paper/2401.13270/) (2024)
- [CCC++: Optimized Color Classified Colorization with Segment Anything Model (SAM) Empowered Object Selective Color Harmonization](https://mlsanity.com/paper/2403.11494/) (2024)

## Abstract (arXiv)

This paper proposes a novel approach to generate multiple color palettes that reflect the semantics of input text and then colorize a given grayscale image according to the generated color palette. In contrast to existing approaches, our model can understand rich text, whether it is a single word, a phrase, or a sentence, and generate multiple possible palettes from it. For this task, we introduce our manually curated dataset called Palette-and-Text (PAT). Our proposed model called Text2Colors consists of two conditional generative adversarial networks: the text-to-palette generation networks and the palette-based colorization networks. The former captures the semantics of the text input and produce relevant color palettes. The latter colorizes a grayscale image using the generated color palette. Our evaluation results show that people preferred our generated palettes over ground truth palettes and that our model can effectively reflect the given palette when colorizing an image.