# Collaboration among Image and Object Level Features for Image Colourisation

arXiv: https://arxiv.org/abs/2101.07576 · HTML: https://mlsanity.com/paper/2101.07576/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 19 Jan 2021.

## Builds on (most cited first, 25 of 25)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Adversarial Feature Learning](https://mlsanity.com/paper/1605.09782/) (2016)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [Real-Time User-Guided Image Colorization with Learned Deep Priors](https://mlsanity.com/paper/1705.02999/) (2017)
- [Deep Exemplar-based Colorization](https://mlsanity.com/paper/1807.06587/) (2018)
- [Instance-aware Image Colorization](https://mlsanity.com/paper/2005.10825/) (2020)
- [Deep Colorization](https://mlsanity.com/paper/1605.00075/) (2016)
- [ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution](https://mlsanity.com/paper/1907.09837/) (2019)
- [Learning Diverse Image Colorization](https://mlsanity.com/paper/1612.01958/) (2016)
- [Fully Automatic Video Colorization with Self-Regularization and Diversity](https://mlsanity.com/paper/1908.01311/) (2019)
- [PixColor: Pixel Recursive Colorization](https://mlsanity.com/paper/1705.07208/) (2017)
- [Data-dependent Initializations of Convolutional Neural Networks](https://mlsanity.com/paper/1511.06856/) (2015)
- [Image Colorization with Generative Adversarial Networks](https://mlsanity.com/paper/1803.05400/) (2018)
- [Pixel-level Semantics Guided Image Colorization](https://mlsanity.com/paper/1808.01597/) (2018)
- [Image Colorization By Capsule Networks](https://mlsanity.com/paper/1908.08307/) (2019)
- [What Do We Understand About Convolutional Networks?](https://mlsanity.com/paper/1803.08834/) (2018)

## Abstract (arXiv)

Image colourisation is an ill-posed problem, with multiple correct solutions which depend on the context and object instances present in the input datum. Previous approaches attacked the problem either by requiring intense user interactions or by exploiting the ability of convolutional neural networks (CNNs) in learning image level (context) features. However, obtaining human hints is not always feasible and CNNs alone are not able to learn object-level semantics unless multiple models pretrained with supervision are considered. In this work, we propose a single network, named UCapsNet, that separate image-level features obtained through convolutions and object-level features captured by means of capsules. Then, by skip connections over different layers, we enforce collaboration between such disentangling factors to produce high quality and plausible image colourisation. We pose the problem as a classification task that can be addressed by a fully self-supervised approach, thus requires no human effort. Experimental results on three benchmark datasets show that our approach outperforms existing methods on standard quality metrics and achieves a state of the art performances on image colourisation. A large scale user study shows that our method is preferred over existing solutions.