# VIIS: Visible and Infrared Information Synthesis for Severe Low-light Image Enhancement

arXiv: https://arxiv.org/abs/2412.13655 · HTML: https://mlsanity.com/paper/2412.13655/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 60 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [EnlightenGAN: Deep Light Enhancement without Paired Supervision](https://mlsanity.com/paper/1906.06972/) (2019)
- [Learning Enriched Features for Real Image Restoration and Enhancement](https://mlsanity.com/paper/2003.06792/) (2020)
- [Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement](https://mlsanity.com/paper/2303.06705/) (2023)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [Implicit Diffusion Models for Continuous Super-Resolution](https://mlsanity.com/paper/2303.16491/) (2023)
- [DenseFuse: A Fusion Approach to Infrared and Visible Images](https://mlsanity.com/paper/1804.08361/) (2018)
- [ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback](https://mlsanity.com/paper/2404.07987/) (2024)
- [DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion](https://mlsanity.com/paper/2303.06840/) (2023)
- [DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation](https://mlsanity.com/paper/2301.03786/) (2023)
- [Low-Light Image Enhancement with Wavelet-based Diffusion Models](https://mlsanity.com/paper/2306.00306/) (2023)
- [RFN-Nest: An end-to-end residual fusion network for infrared and visible images](https://mlsanity.com/paper/2103.04286/) (2021)
- [Equivariant Multi-Modality Image Fusion](https://mlsanity.com/paper/2305.11443/) (2023)
- [LEDNet: Joint Low-light Enhancement and Deblurring in the Dark](https://mlsanity.com/paper/2202.03373/) (2022)
- [Global Structure-Aware Diffusion Process for Low-Light Image Enhancement](https://mlsanity.com/paper/2310.17577/) (2023)
- [Unsupervised Night Image Enhancement: When Layer Decomposition Meets Light-Effects Suppression](https://mlsanity.com/paper/2207.10564/) (2022)
- [Pyramid Diffusion Models For Low-light Image Enhancement](https://mlsanity.com/paper/2305.10028/) (2023)

## Abstract (arXiv)

Images captured in severe low-light circumstances often suffer from significant information absence. Existing singular modality image enhancement methods struggle to restore image regions lacking valid information. By leveraging light-impervious infrared images, visible and infrared image fusion methods have the potential to reveal information hidden in darkness. However, they primarily emphasize inter-modal complementation but neglect intra-modal enhancement, limiting the perceptual quality of output images. To address these limitations, we propose a novel task, dubbed visible and infrared information synthesis (VIIS), which aims to achieve both information enhancement and fusion of the two modalities. Given the difficulty in obtaining ground truth in the VIIS task, we design an information synthesis pretext task (ISPT) based on image augmentation. We employ a diffusion model as the framework and design a sparse attention-based dual-modalities residual (SADMR) conditioning mechanism to enhance information interaction between the two modalities. This mechanism enables features with prior knowledge from both modalities to adaptively and iteratively attend to each modality's information during the denoising process. Our extensive experiments demonstrate that our model qualitatively and quantitatively outperforms not only the state-of-the-art methods in relevant fields but also the newly designed baselines capable of both information enhancement and fusion. The code is available at https://github.com/Chenz418/VIIS.