# ColorMNet: A Memory-based Deep Spatial-Temporal Feature Propagation Network for Video Colorization

arXiv: https://arxiv.org/abs/2404.06251 · HTML: https://mlsanity.com/paper/2404.06251/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 49 references that are in the map.
- First posted on arXiv 9 Apr 2024.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [Real-Time User-Guided Image Colorization with Learned Deep Priors](https://mlsanity.com/paper/1705.02999/) (2017)
- [MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking](https://mlsanity.com/paper/2307.15700/) (2023)
- [Deep Exemplar-based Colorization](https://mlsanity.com/paper/1807.06587/) (2018)
- [Scribbler: Controlling Deep Image Synthesis with Sketch and Color](https://mlsanity.com/paper/1612.00835/) (2016)
- [Instance-aware Image Colorization](https://mlsanity.com/paper/2005.10825/) (2020)
- [Blind Video Temporal Consistency via Deep Video Prior](https://mlsanity.com/paper/2010.11838/) (2020)
- [Deep Exemplar-based Video Colorization](https://mlsanity.com/paper/1906.09909/) (2019)
- [Deep Colorization](https://mlsanity.com/paper/1605.00075/) (2016)
- [Fully Automatic Video Colorization with Self-Regularization and Diversity](https://mlsanity.com/paper/1908.01311/) (2019)
- [DDColor: Towards Photo-Realistic Image Colorization via Dual Decoders](https://mlsanity.com/paper/2212.11613/) (2022)
- [Bringing Old Films Back to Life](https://mlsanity.com/paper/2203.17276/) (2022)
- [Temporally Consistent Video Colorization with Deep Feature Propagation and Self-regularization Learning](https://mlsanity.com/paper/2110.04562/) (2021)
- [Automatic Temporally Coherent Video Colorization](https://mlsanity.com/paper/1904.09527/) (2019)
- [BiSTNet: Semantic Image Prior Guided Bidirectional Temporal Feature Fusion for Deep Exemplar-based Video Colorization](https://mlsanity.com/paper/2212.02268/) (2022)
- [MAMBA: Multi-level Aggregation via Memory Bank for Video Object Detection](https://mlsanity.com/paper/2401.09923/) (2024)
- [VCGAN: Video Colorization with Hybrid Generative Adversarial Network](https://mlsanity.com/paper/2104.12357/) (2021)

## Built on by (most cited first, 8 of 8)

- [Many-for-Many: Unify the Training of Multiple Video and Image Generation and Manipulation Tasks](https://mlsanity.com/paper/2506.01758/) (2025)
- [Everything is a Video: Unifying Modalities through Next-Frame Prediction](https://mlsanity.com/paper/2411.10503/) (2024)
- [Color3D: Controllable and Consistent 3D Colorization with Personalized Colorizer](https://mlsanity.com/paper/2510.10152/) (2025)
- [Consistent Video Colorization via Palette Guidance](https://mlsanity.com/paper/2501.19331/) (2025)
- [Zero-to-Hero: Zero-Shot Initialization Empowering Reference-Based Video Appearance Editing](https://mlsanity.com/paper/2505.23134/) (2025)
- [PhotonSplat: 3D Scene Reconstruction and Colorization from SPAD Sensors](https://mlsanity.com/paper/2506.21680/) (2025)
- [LoGoColor: Local-Global 3D Colorization for 360deg Scenes](https://mlsanity.com/paper/2512.09278/) (2025)
- [VibeFlow: Versatile Video Chroma-Lux Editing through Self-Supervised Learning](https://mlsanity.com/paper/2604.13425/) (2026)

## Abstract (arXiv)

How to effectively explore spatial-temporal features is important for video colorization. Instead of stacking multiple frames along the temporal dimension or recurrently propagating estimated features that will accumulate errors or cannot explore information from far-apart frames, we develop a memory-based feature propagation module that can establish reliable connections with features from far-apart frames and alleviate the influence of inaccurately estimated features. To extract better features from each frame for the above-mentioned feature propagation, we explore the features from large-pretrained visual models to guide the feature estimation of each frame so that the estimated features can model complex scenarios. In addition, we note that adjacent frames usually contain similar contents. To explore this property for better spatial and temporal feature utilization, we develop a local attention module to aggregate the features from adjacent frames in a spatial-temporal neighborhood. We formulate our memory-based feature propagation module, large-pretrained visual model guided feature estimation module, and local attention module into an end-to-end trainable network (named ColorMNet) and show that it performs favorably against state-of-the-art methods on both the benchmark datasets and real-world scenarios. The source code and pre-trained models will be available at urlhttps://github.com/yyang181/colormnet.