# RGB-Multispectral Matching: Dataset, Learning Methodology, Evaluation

arXiv: https://arxiv.org/abs/2206.07047 · HTML: https://mlsanity.com/paper/2206.07047/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 81 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [GA-Net: Guided Aggregation Net for End-to-end Stereo Matching](https://mlsanity.com/paper/1904.06587/) (2019)
- [Group-wise Correlation Stereo Network](https://mlsanity.com/paper/1903.04025/) (2019)
- [CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching](https://mlsanity.com/paper/2104.04314/) (2021)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [Hierarchical Deep Stereo Matching on High-resolution Images](https://mlsanity.com/paper/1912.06704/) (2019)
- [SMD-Nets: Stereo Mixture Density Networks](https://mlsanity.com/paper/2104.03866/) (2021)
- [Domain-invariant Stereo Matching Networks](https://mlsanity.com/paper/1911.13287/) (2019)
- [SegStereo: Exploiting Semantic Information for Disparity Estimation](https://mlsanity.com/paper/1807.11699/) (2018)
- [Real-time self-adaptive deep stereo](https://mlsanity.com/paper/1810.05424/) (2018)
- [Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation](https://mlsanity.com/paper/1808.01838/) (2018)
- [Learning monocular depth estimation infusing traditional stereo knowledge](https://mlsanity.com/paper/1904.04144/) (2019)
- [Cascade Residual Learning: A Two-stage Convolutional Neural Network for Stereo Matching](https://mlsanity.com/paper/1708.09204/) (2017)
- [Reversing the cycle: self-supervised deep stereo through enhanced monocular distillation](https://mlsanity.com/paper/2008.07130/) (2020)
- [EdgeStereo: A Context Integrated Residual Pyramid Network for Stereo Matching](https://mlsanity.com/paper/1803.05196/) (2018)
- [Matching-space Stereo Networks for Cross-domain Generalization](https://mlsanity.com/paper/2010.07347/) (2020)
- [Unsupervised Domain Adaptation for Depth Prediction from Images](https://mlsanity.com/paper/1909.03943/) (2019)
- [Bridging Stereo Matching and Optical Flow via Spatiotemporal Correspondence](https://mlsanity.com/paper/1905.09265/) (2019)
- [Real-Time Semantic Stereo Matching](https://mlsanity.com/paper/1910.00541/) (2019)
- [Continual Adaptation for Deep Stereo](https://mlsanity.com/paper/2007.05233/) (2020)
- [On the confidence of stereo matching in a deep-learning era: a quantitative evaluation](https://mlsanity.com/paper/2101.00431/) (2021)
- [Neural Disparity Refinement for Arbitrary Resolution Stereo](https://mlsanity.com/paper/2110.15367/) (2021)
- [Left-Right Comparative Recurrent Model for Stereo Matching](https://mlsanity.com/paper/1804.00796/) (2018)
- [Detect, Replace, Refine: Deep Structured Prediction For Pixel Wise Labeling](https://mlsanity.com/paper/1612.04770/) (2016)
- [DASC: Robust Dense Descriptor for Multi-modal and Multi-spectral Correspondence Estimation](https://mlsanity.com/paper/1604.07944/) (2016)

## Built on by (most cited first, 6 of 6)

- [XoFTR: Cross-modal Feature Matching Transformer](https://mlsanity.com/paper/2404.09692/) (2024)
- [Cross-Spectral Neural Radiance Fields](https://mlsanity.com/paper/2209.00648/) (2022)
- [A Survey on Deep Stereo Matching in the Twenties](https://mlsanity.com/paper/2407.07816/) (2024)
- [Cross-spectral Gated-RGB Stereo Depth Estimation](https://mlsanity.com/paper/2405.12759/) (2024)
- [Towards RGB-NIR Cross-modality Image Registration and Beyond](https://mlsanity.com/paper/2405.19914/) (2024)
- [Multispectral Demosaicing via Dual Cameras](https://mlsanity.com/paper/2503.22026/) (2025)

## Abstract (arXiv)

We address the problem of registering synchronized color (RGB) and multi-spectral (MS) images featuring very different resolution by solving stereo matching correspondences. Purposely, we introduce a novel RGB-MS dataset framing 13 different scenes in indoor environments and providing a total of 34 image pairs annotated with semi-dense, high-resolution ground-truth labels in the form of disparity maps. To tackle the task, we propose a deep learning architecture trained in a self-supervised manner by exploiting a further RGB camera, required only during training data acquisition. In this setup, we can conveniently learn cross-modal matching in the absence of ground-truth labels by distilling knowledge from an easier RGB-RGB matching task based on a collection of about 11K unlabeled image triplets. Experiments show that the proposed pipeline sets a good performance bar (1.16 pixels average registration error) for future research on this novel, challenging task.