# Implicit Multi-Spectral Transformer: An Lightweight and Effective Visible to Infrared Image Translation Model

arXiv: https://arxiv.org/abs/2404.07072 · HTML: https://mlsanity.com/paper/2404.07072/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 10 Apr 2024.

## Builds on (most cited first, 30 of 32)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Simple Baselines for Image Restoration](https://mlsanity.com/paper/2204.04676/) (2022)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [LLVIP: A Visible-infrared Paired Dataset for Low-light Vision](https://mlsanity.com/paper/2108.10831/) (2021)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [CLIPstyler: Image Style Transfer with a Single Text Condition](https://mlsanity.com/paper/2112.00374/) (2021)
- [Explicit Visual Prompting for Low-Level Structure Segmentations](https://mlsanity.com/paper/2303.10883/) (2023)
- [Multi-Content GAN for Few-Shot Font Style Transfer](https://mlsanity.com/paper/1712.00516/) (2017)
- [Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline](https://mlsanity.com/paper/2204.04120/) (2022)
- [Semantic Image Synthesis via Adversarial Learning](https://mlsanity.com/paper/1707.06873/) (2017)
- [Spatial Fusion GAN for Image Synthesis](https://mlsanity.com/paper/1812.05840/) (2018)
- [Agriculture-Vision: A Large Aerial Image Database for Agricultural Pattern Analysis](https://mlsanity.com/paper/2001.01306/) (2020)
- [BCI: Breast Cancer Immunohistochemical Image Generation through Pyramid Pix2pix](https://mlsanity.com/paper/2204.11425/) (2022)
- [High-Resolution Document Shadow Removal via A Large-Scale Real-World Dataset and A Frequency-Aware Shadow Erasing Net](https://mlsanity.com/paper/2308.14221/) (2023)
- [PTB-TIR: A Thermal Infrared Pedestrian Tracking Benchmark](https://mlsanity.com/paper/1801.05944/) (2018)
- [CoordFill: Efficient High-Resolution Image Inpainting via Parameterized Coordinate Querying](https://mlsanity.com/paper/2303.08524/) (2023)
- [Depth-aware Test-Time Training for Zero-shot Video Object Segmentation](https://mlsanity.com/paper/2403.04258/) (2024)
- [Sem-GAN: Semantically-Consistent Image-to-Image Translation](https://mlsanity.com/paper/1807.04409/) (2018)
- [A Large-scale Film Style Dataset for Learning Multi-frequency Driven Film Enhancement](https://mlsanity.com/paper/2301.08880/) (2023)
- [Brain Diffuser: An End-to-End Brain Image to Brain Network Pipeline](https://mlsanity.com/paper/2303.06410/) (2023)
- [Devignet: High-Resolution Vignetting Removal via a Dual Aggregated Fusion Transformer With Adaptive Channel Expansion](https://mlsanity.com/paper/2308.13739/) (2023)
- [ShaDocNet: Learning Spatial-Aware Tokens in Transformer for Document Shadow Removal](https://mlsanity.com/paper/2211.16675/) (2022)
- [MedPrompt: Cross-Modal Prompting for Multi-Task Medical Image Translation](https://mlsanity.com/paper/2310.02663/) (2023)
- [WavEnhancer: Unifying Wavelet and Transformer for Image Enhancement](https://mlsanity.com/paper/2212.08327/) (2022)

## Built on by (most cited first, 6 of 6)

- [PID: Physics-Informed Diffusion Model for Infrared Image Generation](https://mlsanity.com/paper/2407.09299/) (2024)
- [TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translation](https://mlsanity.com/paper/2602.19430/) (2026)
- [Class-specific diffusion models improve military object detection in a low-data domain](https://mlsanity.com/paper/2604.18076/) (2026)
- [Pix2Next: Leveraging Vision Foundation Models for RGB to NIR Image Translation](https://mlsanity.com/paper/2409.16706/) (2024)
- [Self-DACE++: Robust Low-Light Enhancement via Efficient Adaptive Curve Estimation](https://mlsanity.com/paper/2604.25367/) (2026)
- [Text2Thermal: Physics-Aware Thermal Image Synthesis from Textual Priors](https://mlsanity.com/paper/2609.03585/) (2026)

## Abstract (arXiv)

In the field of computer vision, visible light images often exhibit low contrast in low-light conditions, presenting a significant challenge. While infrared imagery provides a potential solution, its utilization entails high costs and practical limitations. Recent advancements in deep learning, particularly the deployment of Generative Adversarial Networks (GANs), have facilitated the transformation of visible light images to infrared images. However, these methods often experience unstable training phases and may produce suboptimal outputs. To address these issues, we propose a novel end-to-end Transformer-based model that efficiently converts visible light images into high-fidelity infrared images. Initially, the Texture Mapping Module and Color Perception Adapter collaborate to extract texture and color features from the visible light image. The Dynamic Fusion Aggregation Module subsequently integrates these features. Finally, the transformation into an infrared image is refined through the synergistic action of the Color Perception Adapter and the Enhanced Perception Attention mechanism. Comprehensive benchmarking experiments confirm that our model outperforms existing methods, producing infrared images of markedly superior quality, both qualitatively and quantitatively. Furthermore, the proposed model enables more effective downstream applications for infrared images than other methods.