# SA-DNet: A on-demand semantic object registration network adapting to non-rigid deformation

arXiv: https://arxiv.org/abs/2210.09900 · HTML: https://mlsanity.com/paper/2210.09900/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [D2-Net: A Trainable CNN for Joint Detection and Description of Local Features](https://mlsanity.com/paper/1905.03561/) (2019)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [Remote Sensing Image Change Detection with Transformers](https://mlsanity.com/paper/2103.00208/) (2021)
- [R2D2: Repeatable and Reliable Detector and Descriptor](https://mlsanity.com/paper/1906.06195/) (2019)
- [Pixel-Perfect Structure-from-Motion with Featuremetric Refinement](https://mlsanity.com/paper/2108.08291/) (2021)
- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [MatchFormer: Interleaving Attention in Transformers for Feature Matching](https://mlsanity.com/paper/2203.09645/) (2022)
- [Strip Pooling: Rethinking Spatial Pooling for Scene Parsing](https://mlsanity.com/paper/2003.13328/) (2020)
- [RFN-Nest: An end-to-end residual fusion network for infrared and visible images](https://mlsanity.com/paper/2103.04286/) (2021)
- [NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial/Channel Attention Models](https://mlsanity.com/paper/2007.00328/) (2020)
- [Deep Learning for Change Detection in Remote Sensing Images: Comprehensive Review and Meta-Analysis](https://mlsanity.com/paper/2006.05612/) (2020)
- [Robust Registration of Multimodal Remote Sensing Images Based on Structural Similarity](https://mlsanity.com/paper/2103.16871/) (2021)
- [Self-Supervised Depth and Ego-Motion Estimation for Monocular Thermal Video Using Multi-Spectral Consistency Loss](https://mlsanity.com/paper/2103.00760/) (2021)
- [Joint detection and matching of feature points in multimodal images](https://mlsanity.com/paper/1810.12941/) (2018)

## Abstract (arXiv)

As an essential processing step before the fusing of infrared and visible images, the performance of image registration determines whether the two images can be fused at correct spatial position. In the actual scenario, the varied imaging devices may lead to a change in perspective or time gap between shots, making significant non-rigid spatial relationship in infrared and visible images. Even if a large number of feature points are matched, the registration accuracy may still be inadequate, affecting the result of image fusion and other vision tasks. To alleviate this problem, we propose a Semantic-Aware on-Demand registration network (SA-DNet), which mainly purpose is to confine the feature matching process to the semantic region of interest (sROI) by designing semantic-aware module (SAM) and HOL-Deep hybrid matching module (HDM). After utilizing TPS to transform infrared and visible images based on the corresponding feature points in sROI, the registered images are fused using image fusion module (IFM) to achieve a fully functional registration and fusion network. Moreover, we point out that for different demands, this type of approach allows us to select semantic objects for feature matching as needed and accomplishes task-specific registration based on specific requirements. To demonstrate the robustness of SA-DNet for non-rigid distortions, we conduct extensive experiments by comparing SA-DNet with five state-of-the-art infrared and visible image feature matching methods, and the experimental results show that our method adapts better to the presence of non-rigid distortions in the images and provides semantically well-registered images.