# Multi-scale Iterative Residuals for Fast and Scalable Stereo Matching

arXiv: https://arxiv.org/abs/2110.12769 · HTML: https://mlsanity.com/paper/2110.12769/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 25 Oct 2021.

## Builds on (most cited first, 20 of 20)

- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [GA-Net: Guided Aggregation Net for End-to-end Stereo Matching](https://mlsanity.com/paper/1904.06587/) (2019)
- [AANet: Adaptive Aggregation Network for Efficient Stereo Matching](https://mlsanity.com/paper/2004.09548/) (2020)
- [Hierarchical Neural Architecture Search for Deep Stereo Matching](https://mlsanity.com/paper/2010.13501/) (2020)
- [Point-Based Multi-View Stereo Network](https://mlsanity.com/paper/1908.04422/) (2019)
- [Learning Depth with Convolutional Spatial Propagation Network](https://mlsanity.com/paper/1810.02695/) (2018)
- [HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching](https://mlsanity.com/paper/2007.12140/) (2020)
- [DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch](https://mlsanity.com/paper/1909.05845/) (2019)
- [Fast-MVSNet: Sparse-to-Dense Multi-View Stereo With Learned Propagation and Gauss-Newton Refinement](https://mlsanity.com/paper/2003.13017/) (2020)
- [Hierarchical Discrete Distribution Decomposition for Match Density Estimation](https://mlsanity.com/paper/1812.06264/) (2018)
- [StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction](https://mlsanity.com/paper/1807.08865/) (2018)
- [Computing the Stereo Matching Cost with a Convolutional Neural Network](https://mlsanity.com/paper/1409.4326/) (2014)
- [Cascade Residual Learning: A Two-stage Convolutional Neural Network for Stereo Matching](https://mlsanity.com/paper/1708.09204/) (2017)
- [StereoDRNet: Dilated Residual Stereo Net](https://mlsanity.com/paper/1904.02251/) (2019)

## Abstract (arXiv)

Despite the remarkable progress of deep learning in stereo matching, there exists a gap in accuracy between real-time models and slower state-of-the-art models which are suitable for practical applications. This paper presents an iterative multi-scale coarse-to-fine refinement (iCFR) framework to bridge this gap by allowing it to adopt any stereo matching network to make it fast, more efficient and scalable while keeping comparable accuracy. To reduce the computational cost of matching, we use multi-scale warped features to estimate disparity residuals and push the disparity search range in the cost volume to a minimum limit. Finally, we apply a refinement network to recover the loss of precision which is inherent in multi-scale approaches. We test our iCFR framework by adopting the matching networks from state-of-the art GANet and AANet. The result is 49× faster inference time compared to GANetdeep and 4× less memory consumption, with comparable error. Our best performing network, which we call FRSNet is scalable even up to an input resolution of 6K on a GTX 1080Ti, with inference time still below one second and comparable accuracy to AANet+. It out-performs all real-time stereo methods and achieves competitive accuracy on the KITTI benchmark.