# BiSTNet: Semantic Image Prior Guided Bidirectional Temporal Feature Fusion for Deep Exemplar-based Video Colorization

arXiv: https://arxiv.org/abs/2212.02268 · HTML: https://mlsanity.com/paper/2212.02268/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 5 Dec 2022.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://mlsanity.com/paper/2012.02181/) (2020)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [Rethinking Semantic Segmentation: A Prototype View](https://mlsanity.com/paper/2203.15102/) (2022)
- [Real-Time User-Guided Image Colorization with Learned Deep Priors](https://mlsanity.com/paper/1705.02999/) (2017)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [Cascaded Deep Video Deblurring Using Temporal Sharpness Prior](https://mlsanity.com/paper/2004.02501/) (2020)
- [Deep Exemplar-based Colorization](https://mlsanity.com/paper/1807.06587/) (2018)
- [Scribbler: Controlling Deep Image Synthesis with Sketch and Color](https://mlsanity.com/paper/1612.00835/) (2016)
- [Instance-aware Image Colorization](https://mlsanity.com/paper/2005.10825/) (2020)
- [Blind Video Temporal Consistency via Deep Video Prior](https://mlsanity.com/paper/2010.11838/) (2020)
- [Deep Exemplar-based Video Colorization](https://mlsanity.com/paper/1906.09909/) (2019)
- [Deep Colorization](https://mlsanity.com/paper/1605.00075/) (2016)
- [Fully Automatic Video Colorization with Self-Regularization and Diversity](https://mlsanity.com/paper/1908.01311/) (2019)
- [Bringing Old Films Back to Life](https://mlsanity.com/paper/2203.17276/) (2022)
- [Temporally Consistent Video Colorization with Deep Feature Propagation and Self-regularization Learning](https://mlsanity.com/paper/2110.04562/) (2021)
- [Bi-level Feature Alignment for Versatile Image Translation and Manipulation](https://mlsanity.com/paper/2107.03021/) (2021)
- [Automatic Temporally Coherent Video Colorization](https://mlsanity.com/paper/1904.09527/) (2019)
- [Deep Edge-Aware Interactive Colorization against Color-Bleeding Effects](https://mlsanity.com/paper/2107.01619/) (2021)
- [Color2Embed: Fast Exemplar-Based Image Colorization using Color Embeddings](https://mlsanity.com/paper/2106.08017/) (2021)
- [Recurrence-in-Recurrence Networks for Video Deblurring](https://mlsanity.com/paper/2203.06418/) (2022)

## Built on by (most cited first, 13 of 13)

- [ColorMNet: A Memory-based Deep Spatial-Temporal Feature Propagation Network for Video Colorization](https://mlsanity.com/paper/2404.06251/) (2024)
- [Zero-Shot Video Restoration and Enhancement Using Pre-Trained Image Diffusion Model](https://mlsanity.com/paper/2407.01960/) (2024)
- [Diverse Feature Learning by Self-distillation and Reset](https://mlsanity.com/paper/2403.19941/) (2024)
- [Color3D: Controllable and Consistent 3D Colorization with Personalized Colorizer](https://mlsanity.com/paper/2510.10152/) (2025)
- [L-C4: Language-Based Video Colorization for Creative and Consistent Color](https://mlsanity.com/paper/2410.04972/) (2024)
- [Consistent Video Colorization via Palette Guidance](https://mlsanity.com/paper/2501.19331/) (2025)
- [Zero-to-Hero: Zero-Shot Initialization Empowering Reference-Based Video Appearance Editing](https://mlsanity.com/paper/2505.23134/) (2025)
- [MTSIC: Multi-stage Transformer-based GAN for Spectral Infrared Image Colorization](https://mlsanity.com/paper/2506.17540/) (2025)
- [ControlCol: Controllability in Automatic Speaker Video Colorization](https://mlsanity.com/paper/2408.11711/) (2024)
- [Computer-aided Colorization State-of-the-science: A Survey](https://mlsanity.com/paper/2410.02288/) (2024)
- [Tackling Ill-posedness of Reversible Image Conversion with Well-posed Invertible Network](https://mlsanity.com/paper/2508.02111/) (2025)
- [Instant Video Models: Universal Adapters for Stabilizing Image-Based Networks](https://mlsanity.com/paper/2512.03014/) (2025)
- [VibeFlow: Versatile Video Chroma-Lux Editing through Self-Supervised Learning](https://mlsanity.com/paper/2604.13425/) (2026)

## Abstract (arXiv)

How to effectively explore the colors of reference exemplars and propagate them to colorize each frame is vital for exemplar-based video colorization. In this paper, we present an effective BiSTNet to explore colors of reference exemplars and utilize them to help video colorization by a bidirectional temporal feature fusion with the guidance of semantic image prior. We first establish the semantic correspondence between each frame and the reference exemplars in deep feature space to explore color information from reference exemplars. Then, to better propagate the colors of reference exemplars into each frame and avoid the inaccurate matches colors from exemplars we develop a simple yet effective bidirectional temporal feature fusion module to better colorize each frame. We note that there usually exist color-bleeding artifacts around the boundaries of the important objects in videos. To overcome this problem, we further develop a mixed expert block to extract semantic information for modeling the object boundaries of frames so that the semantic image prior can better guide the colorization process for better performance. In addition, we develop a multi-scale recurrent block to progressively colorize frames in a coarse-to-fine manner. Extensive experimental results demonstrate that the proposed BiSTNet performs favorably against state-of-the-art methods on the benchmark datasets. Our code will be made available at urlhttps://yyang181.github.io/BiSTNet/