mlsanityData to 14 Oct 2026

Towards RGB-NIR Cross-modality Image Registration and Beyond

arXiv 2405.19914 ↗First posted 30 May 2024cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

This paper focuses on the area of RGB(visible)-NIR(near-infrared) cross-modality image registration, which is crucial for many downstream vision tasks to fully leverage the complementary information present in visible and infrared images. In this field, researchers face two primary challenges - the absence of a correctly-annotated benchmark with viewpoint variations for evaluating RGB-NIR cross-modality registration methods and the problem of inconsistent local features caused by the appearance discrepancy between RGB-NIR cross-modality images. To address these challenges, we first present the RGB-NIR Image Registration (RGB-NIR-IRegis) benchmark, which, for the first time, enables fair and comprehensive evaluations for the task of RGB-NIR cross-modality image registration. Evaluations of previous methods highlight the significant challenges posed by our RGB-NIR-IRegis benchmark, especially on RGB-NIR image pairs with viewpoint variations. To analyze the causes of the unsatisfying performance, we then design several metrics to reveal the toxic impact of inconsistent local features between visible and infrared images on the model performance. This further motivates us to develop a baseline method named Semantic Guidance Transformer (SGFormer), which utilizes high-level semantic guidance to mitigate the negative impact of local inconsistent features. Despite the simplicity of our motivation, extensive experimental results show the effectiveness of our method.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 57 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

31 of 57 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  4. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  5. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,320
  6. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  7. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  8. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 984
  9. 2017In Defense of the Triplet Loss for Person Re-Identificationcited by 786
  10. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
  11. 2022Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detectioncited by 317
  12. 2017HPatches: A benchmark and evaluation of handcrafted and learned local descriptorscited by 264
19 more
  1. 2021SimSwap: An Efficient Framework For High Fidelity Face Swappingcited by 243
  2. 2019R2D2: Repeatable and Reliable Detector and Descriptorcited by 207
  3. 2021COTR: Correspondence Transformer for Matching Across Imagescited by 183
  4. 2022DKM: Dense Kernelized Feature Matching for Geometry Estimationcited by 177
  5. 2022ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformercited by 171
  6. 2020Patch2Pix: Epipolar-Guided Pixel-Level Correspondencescited by 129
  7. 2020Learning Feature Descriptors using Camera Pose Supervisioncited by 104
  8. 2020Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translationcited by 66
  9. 2017Repeatability Is Not Enough: Learning Affine Regions via Discriminabilitycited by 57
  10. 2018A Dataset and Benchmark for Large-scale Multi-modal Face Anti-spoofingcited by 51
  11. 2022Guide Local Feature Matching by Overlap Estimationcited by 24
  12. 2021FaceController: Controllable Attribute Editing for Face in the Wildcited by 22
  13. 2023DarkVisionNet: Low-Light Imaging via RGB-NIR Fusion with Deep Inconsistency Priorcited by 19
  14. 20223DG-STFM: 3D Geometric Guided Student-Teacher Feature Matchingcited by 14
  15. 2016DASC: Robust Dense Descriptor for Multi-modal and Multi-spectral Correspondence Estimationcited by 10
  16. 2019Unsupervised Cross-spectral Stereo Matching by Learning to Synthesizecited by 8
  17. 2022RGB-Multispectral Matching: Dataset, Learning Methodology, Evaluationcited by 6
  18. 2021There and Back Again: Self-supervised Multispectral Correspondence Estimationcited by 5
  19. 2022Degradation-agnostic Correspondence from Resolution-asymmetric Stereocited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025X-RAFT: Cross-Modal Non-Rigid Registration of Blue and White Light Neurosurgical Hyperspectral Imagescited by 0
  2. 2026Preliminary analysis of RGB-NIR Image Registration techniques for off-road forestry environmentscited by 0