# CapHDR2IR: Caption-Driven Transfer from Visible Light to Infrared Domain

arXiv: https://arxiv.org/abs/2411.16327 · HTML: https://mlsanity.com/paper/2411.16327/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 50 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Builds on (most cited first, 12 of 12)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [DenseCap: Fully Convolutional Localization Networks for Dense Captioning](https://mlsanity.com/paper/1511.07571/) (2015)
- [LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images](https://mlsanity.com/paper/2304.05172/) (2023)
- [Dense Captioning with Joint Inference and Visual Context](https://mlsanity.com/paper/1611.06949/) (2016)
- [PTB-TIR: A Thermal Infrared Pedestrian Tracking Benchmark](https://mlsanity.com/paper/1801.05944/) (2018)
- [Context and Attribute Grounded Dense Captioning](https://mlsanity.com/paper/1904.01410/) (2019)
- [Edge-guided Multi-domain RGB-to-TIR image Translation for Training Vision Tasks with Challenging Labels](https://mlsanity.com/paper/2301.12689/) (2023)
- [Siamese Infrared and Visible Light Fusion Network for RGB-T Tracking](https://mlsanity.com/paper/2103.07302/) (2021)
- [Dynamic Fusion Network for RGBT Tracking](https://mlsanity.com/paper/2109.07662/) (2021)
- [HDRT: A Large-Scale Dataset for Infrared-Guided HDR Imaging](https://mlsanity.com/paper/2406.05475/) (2024)

## Abstract (arXiv)

Infrared (IR) imaging offers advantages in several fields due to its unique ability of capturing content in extreme light conditions. However, the demanding hardware requirements of high-resolution IR sensors limit its widespread application. As an alternative, visible light can be used to synthesize IR images but this causes a loss of fidelity in image details and introduces inconsistencies due to lack of contextual awareness of the scene. This stems from a combination of using visible light with a standard dynamic range, especially under extreme lighting, and a lack of contextual awareness can result in pseudo-thermal-crossover artifacts. This occurs when multiple objects with similar temperatures appear indistinguishable in the training data, further exacerbating the loss of fidelity. To solve this challenge, this paper proposes CapHDR2IR, a novel framework incorporating vision-language models using high dynamic range (HDR) images as inputs to generate IR images. HDR images capture a wider range of luminance variations, ensuring reliable IR image generation in different light conditions. Additionally, a dense caption branch integrates semantic understanding, resulting in more meaningful and discernible IR outputs. Extensive experiments on the HDRT dataset show that the proposed CapHDR2IR achieves state-of-the-art performance compared with existing general domain transfer methods and those tailored for visible-to-infrared image translation.