# Nighttime Thermal Infrared Image Colorization with Feedback-based Object Appearance Learning

arXiv: https://arxiv.org/abs/2310.15688 · HTML: https://mlsanity.com/paper/2310.15688/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 65 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [DRIT++: Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1905.01270/) (2019)
- [ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning](https://mlsanity.com/paper/2007.07936/) (2020)
- [Hierarchical Multi-Scale Attention for Semantic Segmentation](https://mlsanity.com/paper/2005.10821/) (2020)
- [Feedback Network for Image Super-Resolution](https://mlsanity.com/paper/1903.09814/) (2019)
- [Data Augmentation using Random Image Cropping and Patching for Deep CNNs](https://mlsanity.com/paper/1811.09030/) (2018)
- [Context-Aware Mixup for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2108.03557/) (2021)
- [Night-to-Day Image Translation for Retrieval-based Localization](https://mlsanity.com/paper/1809.09767/) (2018)
- [Effective Fusion Factor in FPN for Tiny Object Detection](https://mlsanity.com/paper/2011.02298/) (2020)
- [Segmenting Objects in Day and Night:Edge-Conditioned CNN for Thermal Image Semantic Segmentation](https://mlsanity.com/paper/1907.10303/) (2019)
- [Point-to-Box Network for Accurate Object Detection via Single Point Supervision](https://mlsanity.com/paper/2207.06827/) (2022)
- [See Clearer at Night: Towards Robust Nighttime Semantic Segmentation through Day-Night Image Conversion](https://mlsanity.com/paper/1908.05868/) (2019)
- [Feedback Networks](https://mlsanity.com/paper/1612.09508/) (2016)
- [Sem-GAN: Semantically-Consistent Image-to-Image Translation](https://mlsanity.com/paper/1807.04409/) (2018)
- [Brno Urban Dataset -- The New Data for Self-Driving Agents and Mapping Tasks](https://mlsanity.com/paper/1909.06897/) (2019)
- [SCGAN: Saliency Map-guided Colorization with Generative Adversarial Network](https://mlsanity.com/paper/2011.11377/) (2020)
- [Deep Networks with Internal Selective Attention through Feedback Connections](https://mlsanity.com/paper/1407.3068/) (2014)
- [Thermal Infrared Colorization via Conditional Generative Adversarial Network](https://mlsanity.com/paper/1810.05399/) (2018)
- [Thermal Infrared Image Colorization for Nighttime Driving Scenes with Top-Down Guided Attention](https://mlsanity.com/paper/2104.14374/) (2021)
- [Let There be Light: Improved Traffic Surveillance via Detail Preserving Night-to-Day Transfer](https://mlsanity.com/paper/2105.05011/) (2021)
- [Unpaired Thermal to Visible Spectrum Transfer using Adversarial Training](https://mlsanity.com/paper/1904.02242/) (2019)
- [Memory-Guided Collaborative Attention for Nighttime Thermal Infrared Image Colorization](https://mlsanity.com/paper/2208.02960/) (2022)

## Built on by (most cited first, 2 of 2)

- [TUNI: Unifying Pre-training and Fine-tuning with Modality-Aware Mutual Learning and Rectification for RGB-T Semantic Segmentation](https://mlsanity.com/paper/2509.10005/) (2025)
- [FSCM: Frequency-Enhanced Spatial-Spectral Coupled Mamba for Infrared Hyperspectral Image Colorization](https://mlsanity.com/paper/2605.15880/) (2026)

## Abstract (arXiv)

Stable imaging in adverse environments (e.g., total darkness) makes thermal infrared (TIR) cameras a prevalent option for night scene perception. However, the low contrast and lack of chromaticity of TIR images are detrimental to human interpretation and subsequent deployment of RGB-based vision algorithms. Therefore, it makes sense to colorize the nighttime TIR images by translating them into the corresponding daytime color images (NTIR2DC). Despite the impressive progress made in the NTIR2DC task, how to improve the translation performance of small object classes is under-explored. To address this problem, we propose a generative adversarial network incorporating feedback-based object appearance learning (FoalGAN). Specifically, an occlusion-aware mixup module and corresponding appearance consistency loss are proposed to reduce the context dependence of object translation. As a representative example of small objects in nighttime street scenes, we illustrate how to enhance the realism of traffic light by designing a traffic light appearance loss. To further improve the appearance learning of small objects, we devise a dual feedback learning strategy to selectively adjust the learning frequency of different samples. In addition, we provide pixel-level annotation for a subset of the Brno dataset, which can facilitate the research of NTIR image understanding under multiple weather conditions. Extensive experiments illustrate that the proposed FoalGAN is not only effective for appearance learning of small objects, but also outperforms other image translation methods in terms of semantic preservation and edge consistency for the NTIR2DC task.