# Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches

arXiv: https://arxiv.org/abs/1510.05970 · HTML: https://mlsanity.com/paper/1510.05970/

## Key facts

- Cited by 151 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2015.

## Built on by (most cited first, 30 of 151)

- [Unifying Flow, Stereo and Depth Estimation](https://mlsanity.com/paper/2211.05783/) (2022)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection](https://mlsanity.com/paper/2210.02443/) (2022)
- [SMD-Nets: Stereo Mixture Density Networks](https://mlsanity.com/paper/2104.03866/) (2021)
- [Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats](https://mlsanity.com/paper/2410.12781/) (2024)
- [P3Depth: Monocular Depth Estimation with a Piecewise Planarity Prior](https://mlsanity.com/paper/2204.02091/) (2022)
- [SimpleRecon: 3D Reconstruction Without 3D Convolutions](https://mlsanity.com/paper/2208.14743/) (2022)
- [MobileStereoNet: Towards Lightweight Deep Networks for Stereo Matching](https://mlsanity.com/paper/2108.09770/) (2021)
- [Relational Embedding for Few-Shot Classification](https://mlsanity.com/paper/2108.09666/) (2021)
- [Learning by Aligning: Visible-Infrared Person Re-identification using Cross-Modal Correspondences](https://mlsanity.com/paper/2108.07422/) (2021)
- [Multiview Stereo with Cascaded Epipolar RAFT](https://mlsanity.com/paper/2205.04502/) (2022)
- [Open Challenges in Deep Stereo: the Booster Dataset](https://mlsanity.com/paper/2206.04671/) (2022)
- [MoCha-Stereo: Motif Channel Attention Network for Stereo Matching](https://mlsanity.com/paper/2404.06842/) (2024)
- [Shadow Neural Radiance Fields for Multi-view Satellite Photogrammetry](https://mlsanity.com/paper/2104.09877/) (2021)
- [Correlate-and-Excite: Real-Time Stereo Matching via Guided Cost Volume Excitation](https://mlsanity.com/paper/2108.05773/) (2021)
- [Stereo Anywhere: Robust Zero-Shot Deep Stereo Matching Even Where Either Stereo or Mono Fail](https://mlsanity.com/paper/2412.04472/) (2024)
- [Bilateral Grid Learning for Stereo Matching Networks](https://mlsanity.com/paper/2101.01601/) (2021)
- [NeRF-Supervised Deep Stereo](https://mlsanity.com/paper/2303.17603/) (2023)
- [On Advantages of Mask-level Recognition for Outlier-aware Segmentation](https://mlsanity.com/paper/2301.03407/) (2023)
- [Full Surround Monodepth from Multiple Cameras](https://mlsanity.com/paper/2104.00152/) (2021)
- [Neural Markov Random Field for Stereo Matching](https://mlsanity.com/paper/2403.11193/) (2024)
- [StereoPIFu: Depth Aware Clothed Human Digitization via Stereo Vision](https://mlsanity.com/paper/2104.05289/) (2021)
- [YOLOStereo3D: A Step Back to 2D for Efficient Stereo 3D Detection](https://mlsanity.com/paper/2103.09422/) (2021)
- [Learning Depth Estimation for Transparent and Mirror Surfaces](https://mlsanity.com/paper/2307.15052/) (2023)
- [Booster: a Benchmark for Depth from Images of Specular and Transparent Surfaces](https://mlsanity.com/paper/2301.08245/) (2023)
- [OpenStereo: A Comprehensive Benchmark for Stereo Matching and Strong Baseline](https://mlsanity.com/paper/2312.00343/) (2023)
- [Topologically-Aware Deformation Fields for Single-View 3D Reconstruction](https://mlsanity.com/paper/2205.06267/) (2022)
- [PVStereo: Pyramid Voting Module for End-to-End Self-Supervised Stereo Matching](https://mlsanity.com/paper/2103.07094/) (2021)
- [Learning Camera Localization via Dense Scene Matching](https://mlsanity.com/paper/2103.16792/) (2021)
- [Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo](https://mlsanity.com/paper/2110.05594/) (2021)

## Abstract (arXiv)

We present a method for extracting depth information from a rectified image pair. Our approach focuses on the first stage of many stereo algorithms: the matching cost computation. We approach the problem by learning a similarity measure on small image patches using a convolutional neural network. Training is carried out in a supervised manner by constructing a binary classification data set with examples of similar and dissimilar pairs of patches. We examine two network architectures for this task: one tuned for speed, the other for accuracy. The output of the convolutional neural network is used to initialize the stereo matching cost. A series of post-processing steps follow: cross-based cost aggregation, semiglobal matching, a left-right consistency check, subpixel enhancement, a median filter, and a bilateral filter. We evaluate our method on the KITTI 2012, KITTI 2015, and Middlebury stereo data sets and show that it outperforms other approaches on all three data sets.