# Real-Time User-Guided Image Colorization with Learned Deep Priors

arXiv: https://arxiv.org/abs/1705.02999 · HTML: https://mlsanity.com/paper/1705.02999/

## Key facts

- Cited by 123 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2017.

## Built on by (most cited first, 30 of 123)

- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Editing Conditional Radiance Fields](https://mlsanity.com/paper/2105.06466/) (2021)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [OpenAGI: When LLM Meets Domain Experts](https://mlsanity.com/paper/2304.04370/) (2023)
- [Image-to-Image Translation: Methods and Applications](https://mlsanity.com/paper/2101.08629/) (2021)
- [Colorization Transformer](https://mlsanity.com/paper/2102.04432/) (2021)
- [Expressive Text-to-Image Generation with Rich Text](https://mlsanity.com/paper/2304.06720/) (2023)
- [Open-World Entity Segmentation](https://mlsanity.com/paper/2107.14228/) (2021)
- [High-Resolution Photorealistic Image Translation in Real-Time: A Laplacian Pyramid Translation Network](https://mlsanity.com/paper/2105.09188/) (2021)
- [Using latent space regression to analyze and leverage compositionality in GANs](https://mlsanity.com/paper/2103.10426/) (2021)
- [Towards Vivid and Diverse Image Colorization with Generative Color Prior](https://mlsanity.com/paper/2108.08826/) (2021)
- [Conditional Generation of Audio from Video via Foley Analogies](https://mlsanity.com/paper/2304.08490/) (2023)
- [Instance-wise Hard Negative Example Generation for Contrastive Learning in Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2108.04547/) (2021)
- [Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents](https://mlsanity.com/paper/2402.00798/) (2024)
- [Test-Time Training on Video Streams](https://mlsanity.com/paper/2307.05014/) (2023)
- [Contrastive Representation Learning for Hand Shape Estimation](https://mlsanity.com/paper/2106.04324/) (2021)
- [Posterior-Mean Rectified Flow: Towards Minimum MSE Photo-Realistic Image Restoration](https://mlsanity.com/paper/2410.00418/) (2024)
- [Control Color: Multimodal Diffusion-based Interactive Image Colorization](https://mlsanity.com/paper/2402.10855/) (2024)
- [InstaFormer: Instance-Aware Image-to-Image Translation with Transformer](https://mlsanity.com/paper/2203.16248/) (2022)
- [Cross-Modal 3D Shape Generation and Manipulation](https://mlsanity.com/paper/2207.11795/) (2022)
- [Learning a Sketch Tensor Space for Image Inpainting of Man-made Scenes](https://mlsanity.com/paper/2103.15087/) (2021)
- [Eliminating Gradient Conflict in Reference-based Line-Art Colorization](https://mlsanity.com/paper/2207.06095/) (2022)
- [Diffusing Colors: Image Colorization with Text Guided Diffusion](https://mlsanity.com/paper/2312.04145/) (2023)
- [ViCo: Plug-and-play Visual Condition for Personalized Text-to-image Generation](https://mlsanity.com/paper/2306.00971/) (2023)
- [Temporally Consistent Video Colorization with Deep Feature Propagation and Self-regularization Learning](https://mlsanity.com/paper/2110.04562/) (2021)
- [Image Inpainting with External-internal Learning and Monochromic Bottleneck](https://mlsanity.com/paper/2104.09068/) (2021)
- [Invertible Rescaling Network and Its Extensions](https://mlsanity.com/paper/2210.04188/) (2022)
- [PalGAN: Image Colorization with Palette Generative Adversarial Networks](https://mlsanity.com/paper/2210.11204/) (2022)
- [PixHt-Lab: Pixel Height Based Light Effect Generation for Image Compositing](https://mlsanity.com/paper/2303.00137/) (2023)
- [Improved Diffusion-based Image Colorization via Piggybacked Models](https://mlsanity.com/paper/2304.11105/) (2023)

## Abstract (arXiv)

We propose a deep learning approach for user-guided image colorization. The system directly maps a grayscale image, along with sparse, local user "hints" to an output colorization with a Convolutional Neural Network (CNN). Rather than using hand-defined rules, the network propagates user edits by fusing low-level cues along with high-level semantic information, learned from large-scale data. We train on a million images, with simulated user inputs. To guide the user towards efficient input selection, the system recommends likely colors based on the input image and current user inputs. The colorization is performed in a single feed-forward pass, enabling real-time use. Even with randomly simulated user inputs, we show that the proposed system helps novice users quickly create realistic colorizations, and offers large improvements in colorization quality with just a minute of use. In addition, we demonstrate that the framework can incorporate other user "hints" to the desired colorization, showing an application to color histogram transfer. Our code and models are available at https://richzhang.github.io/ideepcolor.