# All-weather Multi-Modality Image Fusion: Unified Framework and 100k Benchmark

arXiv: https://arxiv.org/abs/2402.02090 · HTML: https://mlsanity.com/paper/2402.02090/

## Key facts

- Cited by 14 papers in the map (data through 7 Oct 2026).
- Builds on 32 of its 91 references that are in the map.
- First posted on arXiv 3 Feb 2024.

## Builds on (most cited first, 30 of 32)

- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Deep Joint Rain Detection and Removal from a Single Image](https://mlsanity.com/paper/1609.07769/) (2016)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [Vision Transformers for Single Image Dehazing](https://mlsanity.com/paper/2204.03883/) (2022)
- [LLVIP: A Visible-infrared Paired Dataset for Low-light Vision](https://mlsanity.com/paper/2108.10831/) (2021)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [Learning A Sparse Transformer Network for Effective Image Deraining](https://mlsanity.com/paper/2303.11950/) (2023)
- [DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion](https://mlsanity.com/paper/2303.06840/) (2023)
- [Multimodal End-to-End Autonomous Driving](https://mlsanity.com/paper/1906.03199/) (2019)
- [Equivariant Multi-Modality Image Fusion](https://mlsanity.com/paper/2305.11443/) (2023)
- [LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images](https://mlsanity.com/paper/2304.05172/) (2023)
- [Curricular Contrastive Regularization for Physics-aware Single Image Dehazing](https://mlsanity.com/paper/2303.14218/) (2023)
- [Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusion](https://mlsanity.com/paper/1910.04066/) (2019)
- [RIDCP: Revitalizing Real Image Dehazing via High-Quality Codebook Priors](https://mlsanity.com/paper/2304.03994/) (2023)
- [MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2303.08600/) (2023)
- [CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion](https://mlsanity.com/paper/2211.10960/) (2022)
- [A Task-guided, Implicitly-searched and Meta-initialized Deep Model for Image Fusion](https://mlsanity.com/paper/2305.15862/) (2023)
- [Toward Real-world Single Image Deraining: A New Benchmark and Beyond](https://mlsanity.com/paper/2206.05514/) (2022)
- [Towards Online Domain Adaptive Object Detection](https://mlsanity.com/paper/2204.05289/) (2022)
- [UCL-Dehaze: Towards Real-world Image Dehazing via Unsupervised Contrastive Learning](https://mlsanity.com/paper/2205.01871/) (2022)
- [Learning a Graph Neural Network with Cross Modality Interaction for Image Fusion](https://mlsanity.com/paper/2308.03256/) (2023)
- [Learned Convolutional Sparse Coding](https://mlsanity.com/paper/1711.00328/) (2017)
- [PAIF: Perception-Aware Infrared-Visible Image Fusion for Attack-Tolerant Semantic Segmentation](https://mlsanity.com/paper/2308.03979/) (2023)
- [U2Net: A General Framework with Spatial-Spectral-Integrated Double U-Net for Image Fusion](https://mlsanity.com/paper/2212.06466/) (2022)
- [Improving Misaligned Multi-modality Image Fusion with One-stage Progressive Dense Registration](https://mlsanity.com/paper/2308.11165/) (2023)
- [Mutual-Guided Dynamic Network for Image Fusion](https://mlsanity.com/paper/2308.12538/) (2023)

## Built on by (most cited first, 14 of 14)

- [ControlFusion: A Controllable Image Fusion Framework with Language-Vision Degradation Prompts](https://mlsanity.com/paper/2503.23356/) (2025)
- [Shuffle Mamba: State Space Models with Random Shuffle for Multi-Modal Image Fusion](https://mlsanity.com/paper/2409.01728/) (2024)
- [AWM-Fuse: Multi-Modality Image Fusion for Adverse Weather via Global and Local Text Perception](https://mlsanity.com/paper/2508.16881/) (2025)
- [MambaTrans: Multimodal Fusion Image Translation via Large Language Model Priors for Downstream Visual Tasks](https://mlsanity.com/paper/2508.07803/) (2025)
- [Text-Guided Channel Perturbation and Pretrained Knowledge Integration for Unified Multi-Modality Image Fusion](https://mlsanity.com/paper/2511.12432/) (2025)
- [Multi-Modal Image Fusion via Intervention-Stable Feature Learning](https://mlsanity.com/paper/2603.23272/) (2026)
- [Breaking Degradation Coupling: A Structural Entropy Guided Decoupled Framework and Benchmark for Infrared Enhancement](https://mlsanity.com/paper/2604.22886/) (2026)
- [CAWM-Mamba: A unified model for infrared-visible image fusion and compound adverse weather restoration](https://mlsanity.com/paper/2603.02560/) (2026)
- [FTPFusion: Frequency-Aware Infrared and Visible Video Fusion with Temporal Perturbation](https://mlsanity.com/paper/2604.01900/) (2026)
- [MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction](https://mlsanity.com/paper/2604.01958/) (2026)
- [Towards In-Context Tone Style Transfer with A Large-Scale Triplet Dataset](https://mlsanity.com/paper/2604.16114/) (2026)
- [Expandable, Compressible, Mineable: Open-World Thermal Image Restoration](https://mlsanity.com/paper/2605.16967/) (2026)
- [Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction](https://mlsanity.com/paper/2606.26812/) (2026)
- [RoES: Rotational Equivariant Selective-frequency Fusion for Multimodal Images](https://mlsanity.com/paper/2609.12497/) (2026)

## Abstract (arXiv)

Multi-modality image fusion (MMIF) combines complementary information from different image modalities to provide a comprehensive and objective interpretation of scenes. However, existing fusion methods cannot resist different weather interferences in real-world scenes, limiting their practical applicability. To bridge this gap, we propose an end-to-end, unified all-weather MMIF model. Rather than focusing solely on pixel-level recovery, our method emphasizes maximizing the representation of key scene information through joint feature fusion and restoration. Specifically, we first decompose images into low-rank and sparse components, enabling effective feature separation for enhanced multi-modality perception. During feature recovery, we introduce a physically-aware clear feature prediction module, inferring variations in light transmission via illumination and reflectance. Clear features generated by the network are used to enhance salient information representation. We also construct a large-scale MMIF dataset with 100,000 image pairs comprehensively across rain, haze, and snow conditions, as well as covering various degradation levels and diverse scenes. Experimental results in both real-world and synthetic scenes demonstrate that the proposed method excels in image fusion and downstream tasks such as object detection, semantic segmentation, and depth estimation. The code is available at https://github.com/ixilai/AWFusion.