# Exemplar-based Video Colorization with Long-term Spatiotemporal Dependency

arXiv: https://arxiv.org/abs/2303.15081 · HTML: https://mlsanity.com/paper/2303.15081/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 62 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [Mobile-Former: Bridging MobileNet and Transformer](https://mlsanity.com/paper/2108.05895/) (2021)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [Tracking Emerges by Colorizing Videos](https://mlsanity.com/paper/1806.09594/) (2018)
- [Conformer: Local Features Coupling Global Representations for Visual Recognition](https://mlsanity.com/paper/2105.03889/) (2021)
- [Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks](https://mlsanity.com/paper/1810.12348/) (2018)
- [Colorization Transformer](https://mlsanity.com/paper/2102.04432/) (2021)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [Deep Exemplar-based Colorization](https://mlsanity.com/paper/1807.06587/) (2018)
- [Deep Exemplar-based Video Colorization](https://mlsanity.com/paper/1906.09909/) (2019)
- [ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution](https://mlsanity.com/paper/1907.09837/) (2019)
- [Video Propagation Networks](https://mlsanity.com/paper/1612.05478/) (2016)
- [Learning Image Matching by Simply Watching Video](https://mlsanity.com/paper/1603.06041/) (2016)

## Built on by (most cited first, 4 of 4)

- [ColorMNet: A Memory-based Deep Spatial-Temporal Feature Propagation Network for Video Colorization](https://mlsanity.com/paper/2404.06251/) (2024)
- [Temporal Consistent Automatic Video Colorization via Semantic Correspondence](https://mlsanity.com/paper/2305.07904/) (2023)
- [Consistent Video Colorization via Palette Guidance](https://mlsanity.com/paper/2501.19331/) (2025)
- [Computer-aided Colorization State-of-the-science: A Survey](https://mlsanity.com/paper/2410.02288/) (2024)

## Abstract (arXiv)

Exemplar-based video colorization is an essential technique for applications like old movie restoration. Although recent methods perform well in still scenes or scenes with regular movement, they always lack robustness in moving scenes due to their weak ability in modeling long-term dependency both spatially and temporally, leading to color fading, color discontinuity or other artifacts. To solve this problem, we propose an exemplar-based video colorization framework with long-term spatiotemporal dependency. To enhance the long-term spatial dependency, a parallelized CNN-Transformer block and a double head non-local operation are designed. The proposed CNN-Transformer block can better incorporate long-term spatial dependency with local texture and structural features, and the double head non-local operation further leverages the performance of augmented feature. While for long-term temporal dependency enhancement, we further introduce the novel linkage subnet. The linkage subnet propagate motion information across adjacent frame blocks and help to maintain temporal continuity. Experiments demonstrate that our model outperforms recent state-of-the-art methods both quantitatively and qualitatively. Also, our model can generate more colorful, realistic and stabilized results, especially for scenes where objects change greatly and irregularly.