# CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion

arXiv: https://arxiv.org/abs/2211.10960 · HTML: https://mlsanity.com/paper/2211.10960/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 106 references that are in the map.
- First posted on arXiv 20 Nov 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [Retinex-inspired Unrolling with Cooperative Prior Architecture Search for Low-light Image Enhancement](https://mlsanity.com/paper/2012.05609/) (2020)
- [Residual Non-local Attention Networks for Image Restoration](https://mlsanity.com/paper/1903.10082/) (2019)
- [Contrastive Learning for Compact Single Image Dehazing](https://mlsanity.com/paper/2104.09367/) (2021)
- [A Decomposable Attention Model for Natural Language Inference](https://mlsanity.com/paper/1606.01933/) (2016)
- [DetCo: Unsupervised Contrastive Learning for Object Detection](https://mlsanity.com/paper/2102.04803/) (2021)
- [DenseFuse: A Fusion Approach to Infrared and Visible Images](https://mlsanity.com/paper/1804.08361/) (2018)
- [Expectation-Maximization Attention Networks for Semantic Segmentation](https://mlsanity.com/paper/1907.13426/) (2019)
- [RFN-Nest: An end-to-end residual fusion network for infrared and visible images](https://mlsanity.com/paper/2103.04286/) (2021)
- [Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration](https://mlsanity.com/paper/2205.11876/) (2022)
- [DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2003.09210/) (2020)
- [Attention Guided Low-light Image Enhancement with a Large Scale Low-light Simulation Dataset](https://mlsanity.com/paper/1908.00682/) (2019)
- [Bilateral Attention Network for RGB-D Salient Object Detection](https://mlsanity.com/paper/2004.14582/) (2020)
- [Infrared and Visible Image Fusion using a Deep Learning Framework](https://mlsanity.com/paper/1804.06992/) (2018)
- [Blind Image Super-Resolution via Contrastive Representation Learning](https://mlsanity.com/paper/2107.00708/) (2021)
- [Image Fusion Transformer](https://mlsanity.com/paper/2107.09011/) (2021)
- [Learning with Nested Scene Modeling and Cooperative Architecture Search for Low-Light Vision](https://mlsanity.com/paper/2112.04719/) (2021)
- [Learning Deformable Image Registration from Optimization: Perspective, Modules, Bilevel Training and Beyond](https://mlsanity.com/paper/2004.14557/) (2020)
- [Bayesian Fusion for Infrared and Visible Images](https://mlsanity.com/paper/2005.05839/) (2020)
- [Automated Learning for Deformable Medical Image Registration by Jointly Optimizing Network Architectures and Objective Functions](https://mlsanity.com/paper/2203.06810/) (2022)

## Built on by (most cited first, 30 of 63)

- [Equivariant Multi-Modality Image Fusion](https://mlsanity.com/paper/2305.11443/) (2023)
- [FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba](https://mlsanity.com/paper/2404.09498/) (2024)
- [A Task-guided, Implicitly-searched and Meta-initialized Deep Model for Image Fusion](https://mlsanity.com/paper/2305.15862/) (2023)
- [Infrared and Visible Image Fusion: From Data Compatibility to Task Adaption](https://mlsanity.com/paper/2501.10761/) (2025)
- [An Interactively Reinforced Paradigm for Joint Infrared-Visible Image Fusion and Saliency Object Detection](https://mlsanity.com/paper/2305.09999/) (2023)
- [Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond](https://mlsanity.com/paper/2503.01210/) (2025)
- [DCEvo: Discriminative Cross-Dimensional Evolutionary Learning for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2503.17673/) (2025)
- [One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion](https://mlsanity.com/paper/2502.19854/) (2025)
- [Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model](https://mlsanity.com/paper/2410.23905/) (2024)
- [Task-driven Image Fusion with Learnable Fusion Loss](https://mlsanity.com/paper/2412.03240/) (2024)
- [ReFusion: Learning Image Fusion from Reconstruction with Learnable Loss via Meta-Learning](https://mlsanity.com/paper/2312.07943/) (2023)
- [Efficient Rectified Flow for Image Fusion](https://mlsanity.com/paper/2509.16549/) (2025)
- [All-weather Multi-Modality Image Fusion: Unified Framework and 100k Benchmark](https://mlsanity.com/paper/2402.02090/) (2024)
- [Conditional Controllable Image Fusion](https://mlsanity.com/paper/2411.01573/) (2024)
- [Unified-modal Salient Object Detection via Adaptive Prompt Learning](https://mlsanity.com/paper/2311.16835/) (2023)
- [UMCFuse: A Unified Multiple Complex Scenes Infrared and Visible Image Fusion Framework](https://mlsanity.com/paper/2402.02096/) (2024)
- [Instruction-Driven Fusion of Infrared-Visible Images: Tailoring for Diverse Downstream Tasks](https://mlsanity.com/paper/2411.09387/) (2024)
- [HUPE: Heuristic Underwater Perceptual Enhancement with Semantic Collaborative Learning](https://mlsanity.com/paper/2411.18296/) (2024)
- [DEAL: Data-Efficient Adversarial Learning for High-Quality Infrared Imaging](https://mlsanity.com/paper/2503.00905/) (2025)
- [Towards Robust Image Stitching: An Adaptive Resistance Learning against Compatible Attacks](https://mlsanity.com/paper/2402.15959/) (2024)
- [Part-Whole Relational Fusion Towards Multi-Modal Scene Understanding](https://mlsanity.com/paper/2410.14944/) (2024)
- [Contourlet Refinement Gate Framework for Thermal Spectrum Distribution Regularized Infrared Image Super-Resolution](https://mlsanity.com/paper/2411.12530/) (2024)
- [UniFusion: A Unified Image Fusion Framework with Robust Representation and Source-Aware Preservation](https://mlsanity.com/paper/2603.14214/) (2026)
- [Practical exposure correction via compensation](https://mlsanity.com/paper/2212.14245/) (2022)
- [WaterFlow: Heuristic Normalizing Flow for Underwater Image Enhancement and Beyond](https://mlsanity.com/paper/2308.00931/) (2023)
- [Fusion of Single and Integral Multispectral Aerial Images](https://mlsanity.com/paper/2311.17515/) (2023)
- [Task-Generalized Adaptive Cross-Domain Learning for Multimodal Image Fusion](https://mlsanity.com/paper/2508.15505/) (2025)
- [Holistic Dynamic Frequency Transformer for Image Fusion and Exposure Correction](https://mlsanity.com/paper/2309.01183/) (2023)
- [IAIFNet: An Illumination-Aware Infrared and Visible Image Fusion Network](https://mlsanity.com/paper/2309.14997/) (2023)
- [Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond](https://mlsanity.com/paper/2410.12274/) (2024)

## Abstract (arXiv)

Infrared and visible image fusion targets to provide an informative image by combining complementary information from different sensors. Existing learning-based fusion approaches attempt to construct various loss functions to preserve complementary features, while neglecting to discover the inter-relationship between the two modalities, leading to redundant or even invalid information on the fusion results. Moreover, most methods focus on strengthening the network with an increase in depth while neglecting the importance of feature transmission, causing vital information degeneration. To alleviate these issues, we propose a coupled contrastive learning network, dubbed CoCoNet, to realize infrared and visible image fusion in an end-to-end manner. Concretely, to simultaneously retain typical features from both modalities and to avoid artifacts emerging on the fused result, we develop a coupled contrastive constraint in our loss function. In a fused image, its foreground target / background detail part is pulled close to the infrared / visible source and pushed far away from the visible / infrared source in the representation space. We further exploit image characteristics to provide data-sensitive weights, allowing our loss function to build a more reliable relationship with source images. A multi-level attention module is established to learn rich hierarchical feature representation and to comprehensively transfer features in the fusion process. We also apply the proposed CoCoNet on medical image fusion of different types, e.g., magnetic resonance image, positron emission tomography image, and single photon emission computed tomography image. Extensive experiments demonstrate that our method achieves state-of-the-art (SOTA) performance under both subjective and objective evaluation, especially in preserving prominent targets and recovering vital textural details.