# Diffusing Colors: Image Colorization with Text Guided Diffusion

arXiv: https://arxiv.org/abs/2312.04145 · HTML: https://mlsanity.com/paper/2312.04145/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [Towards Real-World Blind Face Restoration with Generative Facial Prior](https://mlsanity.com/paper/2101.04061/) (2021)
- [SpaText: Spatio-Textual Representation for Controllable Image Generation](https://mlsanity.com/paper/2211.14305/) (2022)
- [Real-Time User-Guided Image Colorization with Learned Deep Priors](https://mlsanity.com/paper/1705.02999/) (2017)
- [Instance-aware Image Colorization](https://mlsanity.com/paper/2005.10825/) (2020)
- [ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution](https://mlsanity.com/paper/1907.09837/) (2019)
- [Towards Vivid and Diverse Image Colorization with Generative Color Prior](https://mlsanity.com/paper/2108.08826/) (2021)
- [BigColor: Colorization using a Generative Color Prior for Natural Images](https://mlsanity.com/paper/2207.09685/) (2022)
- [Learning to Color from Language](https://mlsanity.com/paper/1804.06026/) (2018)
- [Improved Diffusion-based Image Colorization via Piggybacked Models](https://mlsanity.com/paper/2304.11105/) (2023)

## Built on by (most cited first, 25 of 25)

- [Control Color: Multimodal Diffusion-based Interactive Image Colorization](https://mlsanity.com/paper/2402.10855/) (2024)
- [Group Diffusion Transformers are Unsupervised Multitask Learners](https://mlsanity.com/paper/2410.15027/) (2024)
- [MangaNinja: Line Art Colorization with Precise Reference Following](https://mlsanity.com/paper/2501.08332/) (2025)
- [Affective Image Editing: Shaping Emotional Factors via Text Descriptions](https://mlsanity.com/paper/2505.18699/) (2025)
- [Follow-Your-Color: Multi-Instance Sketch Colorization](https://mlsanity.com/paper/2503.16948/) (2025)
- [Tiled Diffusion](https://mlsanity.com/paper/2412.15185/) (2024)
- [C-DiffSET: Leveraging Latent Diffusion for SAR-to-EO Image Translation with Confidence-Guided Reliable Object Generation](https://mlsanity.com/paper/2411.10788/) (2024)
- [ColorFlow: Retrieval-Augmented Image Sequence Colorization](https://mlsanity.com/paper/2412.11815/) (2024)
- [Cobra: Efficient Line Art COlorization with BRoAder References](https://mlsanity.com/paper/2504.12240/) (2025)
- [Image Referenced Sketch Colorization Based on Animation Creation Workflow](https://mlsanity.com/paper/2502.19937/) (2025)
- [Exploring Palette based Color Guidance in Diffusion Models](https://mlsanity.com/paper/2508.08754/) (2025)
- [From Fragment to One Piece: A Survey on AI-Driven Graphic Design](https://mlsanity.com/paper/2503.18641/) (2025)
- [Color3D: Controllable and Consistent 3D Colorization with Personalized Colorizer](https://mlsanity.com/paper/2510.10152/) (2025)
- [Language-based Image Colorization: A Benchmark and Beyond](https://mlsanity.com/paper/2503.14974/) (2025)
- [PRevivor: Reviving Ancient Chinese Paintings using Prior-Guided Color Transformers](https://mlsanity.com/paper/2511.01274/) (2025)
- [Controllable 3D Object Generation with Single Image Prompt](https://mlsanity.com/paper/2511.22194/) (2025)
- [SketchDeco: Training-Free Latent Composition for Precise Sketch Colourisation](https://mlsanity.com/paper/2405.18716/) (2024)
- [MRStyle: A Unified Framework for Color Style Transfer with Multi-Modality Reference](https://mlsanity.com/paper/2409.05250/) (2024)
- [Computer-aided Colorization State-of-the-science: A Survey](https://mlsanity.com/paper/2410.02288/) (2024)
- [Instance-aware Image Colorization with Controllable Textual Descriptions and Segmentation Masks](https://mlsanity.com/paper/2505.08705/) (2025)
- [Harnessing the Power of Training-Free Techniques in Text-to-2D Generation for Text-to-3D Generation via Score Distillation Sampling](https://mlsanity.com/paper/2505.19868/) (2025)
- [Multimodal Image Colorization: Quantifying the Impact of Text-Conditioned Guidance on Grayscale-to-Color Translation](https://mlsanity.com/paper/2606.20722/) (2026)
- [Reviving Ancient Paintings via Poem: A Colorization Framework for Aligning Cultural Semantics](https://mlsanity.com/paper/2607.17638/) (2026)
- [Beyond Fixed Luminance: Towards Panchromatic and Orthochromatic Image Colorization](https://mlsanity.com/paper/2608.10798/) (2026)
- [MCIQA-2K: A Multi-Dimensional Dataset and No-Reference Quality Assessment Benchmark for Colorized Images](https://mlsanity.com/paper/2609.14495/) (2026)

## Abstract (arXiv)

The colorization of grayscale images is a complex and subjective task with significant challenges. Despite recent progress in employing large-scale datasets with deep neural networks, difficulties with controllability and visual quality persist. To tackle these issues, we present a novel image colorization framework that utilizes image diffusion techniques with granular text prompts. This integration not only produces colorization outputs that are semantically appropriate but also greatly improves the level of control users have over the colorization process. Our method provides a balance between automation and control, outperforming existing techniques in terms of visual quality and semantic coherence. We leverage a pretrained generative Diffusion Model, and show that we can finetune it for the colorization task without losing its generative power or attention to text prompts. Moreover, we present a novel CLIP-based ranking model that evaluates color vividness, enabling automatic selection of the most suitable level of vividness based on the specific scene semantics. Our approach holds potential particularly for color enhancement and historical image colorization.