# Improving Misaligned Multi-modality Image Fusion with One-stage Progressive Dense Registration

arXiv: https://arxiv.org/abs/2308.11165 · HTML: https://mlsanity.com/paper/2308.11165/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 62 references that are in the map.
- First posted on arXiv 22 Aug 2023.

## Builds on (most cited first, 28 of 28)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [DenseFuse: A Fusion Approach to Infrared and Visible Images](https://mlsanity.com/paper/1804.08361/) (2018)
- [DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion](https://mlsanity.com/paper/2303.06840/) (2023)
- [An Unsupervised Learning Model for Deformable Medical Image Registration](https://mlsanity.com/paper/1802.02604/) (2018)
- [GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences](https://mlsanity.com/paper/1912.05524/) (2019)
- [Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation](https://mlsanity.com/paper/2308.02097/) (2023)
- [RFN-Nest: An end-to-end residual fusion network for infrared and visible images](https://mlsanity.com/paper/2103.04286/) (2021)
- [LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images](https://mlsanity.com/paper/2304.05172/) (2023)
- [Practical Blind Image Denoising via Swin-Conv-UNet and Data Synthesis](https://mlsanity.com/paper/2203.13278/) (2022)
- [Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration](https://mlsanity.com/paper/2205.11876/) (2022)
- [DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2003.09210/) (2020)
- [DGC-Net: Dense Geometric Correspondence Network](https://mlsanity.com/paper/1810.08393/) (2018)
- [Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation](https://mlsanity.com/paper/2003.08073/) (2020)
- [A convex formulation for hyperspectral image superresolution via subspace-based regularization](https://mlsanity.com/paper/1411.4005/) (2014)
- [An Interactively Reinforced Paradigm for Joint Infrared-Visible Image Fusion and Saliency Object Detection](https://mlsanity.com/paper/2305.09999/) (2023)
- [Deep Learning based Inter-Modality Image Registration Supervised by Intra-Modality Similarity](https://mlsanity.com/paper/1804.10735/) (2018)
- [Fast and High-Quality Blind Multi-Spectral Image Pansharpening](https://mlsanity.com/paper/2103.09943/) (2021)
- [PPT Fusion: Pyramid Patch Transformerfor a Case Study in Image Fusion](https://mlsanity.com/paper/2107.13967/) (2021)

## Built on by (most cited first, 17 of 17)

- [All-weather Multi-Modality Image Fusion: Unified Framework and 100k Benchmark](https://mlsanity.com/paper/2402.02090/) (2024)
- [Unified-modal Salient Object Detection via Adaptive Prompt Learning](https://mlsanity.com/paper/2311.16835/) (2023)
- [BSAFusion: A Bidirectional Stepwise Feature Alignment Network for Unaligned Medical Image Fusion](https://mlsanity.com/paper/2412.08050/) (2024)
- [Task-Generalized Adaptive Cross-Domain Learning for Multimodal Image Fusion](https://mlsanity.com/paper/2508.15505/) (2025)
- [SeqCSIST: Sequential Closely-Spaced Infrared Small Target Unmixing](https://mlsanity.com/paper/2507.09556/) (2025)
- [Encoder-Only Image Registration](https://mlsanity.com/paper/2509.00451/) (2025)
- [Single-image reflection removal via self-supervised diffusion models](https://mlsanity.com/paper/2412.20466/) (2024)
- [UniReg: Conditional Unified Model for Medical Image Registration](https://mlsanity.com/paper/2503.12868/) (2025)
- [How Do Optical Flow and Textual Prompts Collaborate to Assist in Audio-Visual Semantic Segmentation?](https://mlsanity.com/paper/2601.08133/) (2026)
- [FusionRegister: Every Infrared and Visible Image Fusion Deserves Registration](https://mlsanity.com/paper/2603.07667/) (2026)
- [Bi-directional Self-Registration for Misaligned Infrared-Visible Image Fusion](https://mlsanity.com/paper/2505.06920/) (2025)
- [Hyperbolic Cycle Alignment for Infrared-Visible Image Fusion](https://mlsanity.com/paper/2507.23508/) (2025)
- [DSCSNet: A Dynamic Sparse Compression Sensing Network for Closely-Spaced Infrared Small Target Unmixing](https://mlsanity.com/paper/2603.21192/) (2026)
- [Uncertainty-aware Spatial-Frequency Registration and Fusion for Infrared and Visible Images](https://mlsanity.com/paper/2605.13049/) (2026)
- [BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion](https://mlsanity.com/paper/2607.24110/) (2026)
- [XCalib Depth-Guided Geometric Optimization for Dense Thermal-Visible Video Registration](https://mlsanity.com/paper/2609.21770/) (2026)
- [SAGE: Source-Anchored Guidance via Frequency Equalization for Hierarchical RGB-T Alignment and Fusion](https://mlsanity.com/paper/2609.30703/) (2026)

## Abstract (arXiv)

Misalignments between multi-modality images pose challenges in image fusion, manifesting as structural distortions and edge ghosts. Existing efforts commonly resort to registering first and fusing later, typically employing two cascaded stages for registration,i.e., coarse registration and fine registration. Both stages directly estimate the respective target deformation fields. In this paper, we argue that the separated two-stage registration is not compact, and the direct estimation of the target deformation fields is not accurate enough. To address these challenges, we propose a Cross-modality Multi-scale Progressive Dense Registration (C-MPDR) scheme, which accomplishes the coarse-to-fine registration exclusively using a one-stage optimization, thus improving the fusion performance of misaligned multi-modality images. Specifically, two pivotal components are involved, a dense Deformation Field Fusion (DFF) module and a Progressive Feature Fine (PFF) module. The DFF aggregates the predicted multi-scale deformation sub-fields at the current scale, while the PFF progressively refines the remaining misaligned features. Both work together to accurately estimate the final deformation fields. In addition, we develop a Transformer-Conv-based Fusion (TCF) subnetwork that considers local and long-range feature dependencies, allowing us to capture more informative features from the registered infrared and visible images for the generation of high-quality fused images. Extensive experimental analysis demonstrates the superiority of the proposed method in the fusion of misaligned cross-modality images.