# Visible and infrared self-supervised fusion trained on a single example

arXiv: https://arxiv.org/abs/2307.04100 · HTML: https://mlsanity.com/paper/2307.04100/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 9 Jul 2023.

## Builds on (most cited first, 15 of 15)

- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- ["Zero-Shot" Super-Resolution using Deep Internal Learning](https://mlsanity.com/paper/1712.06087/) (2017)
- [U-Net and its variants for medical image segmentation: theory and applications](https://mlsanity.com/paper/2011.01118/) (2020)
- [NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial/Channel Attention Models](https://mlsanity.com/paper/2007.00328/) (2020)
- [Challenge-Aware RGBT Tracking](https://mlsanity.com/paper/2007.13143/) (2020)
- [Infrared and Visible Image Fusion using a Deep Learning Framework](https://mlsanity.com/paper/1804.06992/) (2018)
- [Joint detection and matching of feature points in multimodal images](https://mlsanity.com/paper/1810.12941/) (2018)
- [Deep Multi-Spectral Registration Using Invariant Descriptor Learning](https://mlsanity.com/paper/1801.05171/) (2018)
- [The Level Weighted Structural Similarity Loss: A Step Away from the MSE](https://mlsanity.com/paper/1904.13362/) (2019)
- [Multi-scale Processing of Noisy Images using Edge Preservation Losses](https://mlsanity.com/paper/1803.09420/) (2018)
- [Registration and Fusion of Multi-Spectral Images Using a Novel Edge Descriptor](https://mlsanity.com/paper/1711.01543/) (2017)
- [Multispectral image fusion based on super pixel segmentation](https://mlsanity.com/paper/2112.11329/) (2021)

## Abstract (arXiv)

Multispectral imaging is an important task of image processing and computer vision, which is especially relevant to applications such as dehazing or object detection. With the development of the RGBT (RGB & Thermal) sensor, the problem of visible (RGB) to Near Infrared (NIR) image fusion has become particularly timely. Indeed, while visible images see color, but suffer from noise, haze, and clouds, the NIR channel captures a clearer picture. The proposed approach fuses these two channels by training a Convolutional Neural Network by Self Supervised Learning (SSL) on a single example. For each such pair, RGB and NIR, the network is trained for seconds to deduce the final fusion. The SSL is based on the comparison of the Structure of Similarity and Edge-Preservation losses, where the labels for the SSL are the input channels themselves. This fusion preserves the relevant detail of each spectral channel without relying on a heavy training process. Experiments demonstrate that the proposed approach achieves similar or better qualitative and quantitative multispectral fusion results than other state-of-the-art methods that do not rely on heavy training and/or large datasets.