# Rethinking the Key Factors for the Generalization of Remote Sensing Stereo Matching Networks

arXiv: https://arxiv.org/abs/2408.07613 · HTML: https://mlsanity.com/paper/2408.07613/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 14 Aug 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Domain Generalization with MixStyle](https://mlsanity.com/paper/2104.02008/) (2021)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation](https://mlsanity.com/paper/2203.11483/) (2022)
- [Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2011.02910/) (2020)
- [CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching](https://mlsanity.com/paper/2104.04314/) (2021)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [Domain-invariant Stereo Matching Networks](https://mlsanity.com/paper/1911.13287/) (2019)
- [Revisiting Domain Generalized Stereo Matching Networks from a Feature Consistency Perspective](https://mlsanity.com/paper/2203.10887/) (2022)
- [Parallax Attention for Unsupervised Stereo Correspondence Learning](https://mlsanity.com/paper/2009.08250/) (2020)
- [ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks](https://mlsanity.com/paper/2201.02263/) (2022)
- [Matching-space Stereo Networks for Cross-domain Generalization](https://mlsanity.com/paper/2010.07347/) (2020)
- [Guided Stereo Matching](https://mlsanity.com/paper/1905.10107/) (2019)
- [Zoom and Learn: Generalizing Deep Stereo Matching to Novel Domains](https://mlsanity.com/paper/1803.06641/) (2018)
- [A New Stereo Benchmarking Dataset for Satellite Images](https://mlsanity.com/paper/1907.04404/) (2019)
- [WHU-Stereo: A Challenging Benchmark for Stereo Matching of High-Resolution Satellite Images](https://mlsanity.com/paper/2206.02342/) (2022)
- [Improving Deep Stereo Network Generalization with Geometric Priors](https://mlsanity.com/paper/2008.11098/) (2020)

## Abstract (arXiv)

Stereo matching, a critical step of 3D reconstruction, has fully shifted towards deep learning due to its strong feature representation of remote sensing images. However, ground truth for stereo matching task relies on expensive airborne LiDAR data, thus making it difficult to obtain enough samples for supervised learning. To improve the generalization ability of stereo matching networks on cross-domain data from different sensors and scenarios, in this paper, we dedicate to study key training factors from three perspectives. (1) For the selection of training dataset, it is important to select data with similar regional target distribution as the test set instead of utilizing data from the same sensor. (2) For model structure, cascaded structure that flexibly adapts to different sizes of features is preferred. (3) For training manner, unsupervised methods generalize better than supervised methods, and we design an unsupervised early-stop strategy to help retain the best model with pre-trained weights as the basis. Extensive experiments are conducted to support the previous findings, on the basis of which we present an unsupervised stereo matching network with good generalization performance. We release the source code and the datasets at https://github.com/Elenairene/RKF_RSSM to reproduce the results and encourage future work.