# Automatic Controllable Colorization via Imagination

arXiv: https://arxiv.org/abs/2404.05661 · HTML: https://mlsanity.com/paper/2404.05661/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 95 references that are in the map.
- First posted on arXiv 8 Apr 2024.

## Builds on (most cited first, 30 of 58)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [On Fast Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2106.00132/) (2021)
- [You Only Need Adversarial Supervision for Semantic Image Synthesis](https://mlsanity.com/paper/2012.04781/) (2020)

## Built on by (most cited first, 7 of 7)

- [ColorFLUX: A Structure-Color Decoupling Framework for Old Photo Colorization](https://mlsanity.com/paper/2603.28162/) (2026)
- [Consistent Video Colorization via Palette Guidance](https://mlsanity.com/paper/2501.19331/) (2025)
- [Language-based Image Colorization: A Benchmark and Beyond](https://mlsanity.com/paper/2503.14974/) (2025)
- [Art3D: Training-Free 3D Generation from Flat-Colored Illustration](https://mlsanity.com/paper/2504.10466/) (2025)
- [VADMamba++: Efficient Video Anomaly Detection via Hybrid Modeling in Grayscale Space](https://mlsanity.com/paper/2604.00360/) (2026)
- [SketchDeco: Training-Free Latent Composition for Precise Sketch Colourisation](https://mlsanity.com/paper/2405.18716/) (2024)
- [Paint-Anything: Unified Any-Color Control for Image Generation and Editing](https://mlsanity.com/paper/2609.20816/) (2026)

## Abstract (arXiv)

We propose a framework for automatic colorization that allows for iterative editing and modifications. The core of our framework lies in an imagination module: by understanding the content within a grayscale image, we utilize a pre-trained image generation model to generate multiple images that contain the same content. These images serve as references for coloring, mimicking the process of human experts. As the synthesized images can be imperfect or different from the original grayscale image, we propose a Reference Refinement Module to select the optimal reference composition. Unlike most previous end-to-end automatic colorization algorithms, our framework allows for iterative and localized modifications of the colorization results because we explicitly model the coloring samples. Extensive experiments demonstrate the superiority of our framework over existing automatic colorization algorithms in editability and flexibility. Project page: https://xy-cong.github.io/imagine-colorization.