# Real-time self-adaptive deep stereo

arXiv: https://arxiv.org/abs/1810.05424 · HTML: https://mlsanity.com/paper/1810.05424/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2018.

## Built on by (most cited first, 30 of 80)

- [CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching](https://mlsanity.com/paper/2104.04314/) (2021)
- [SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation](https://mlsanity.com/paper/2206.08367/) (2022)
- [SMD-Nets: Stereo Mixture Density Networks](https://mlsanity.com/paper/2104.03866/) (2021)
- [Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks](https://mlsanity.com/paper/2105.12374/) (2021)
- [Open Challenges in Deep Stereo: the Booster Dataset](https://mlsanity.com/paper/2206.04671/) (2022)
- [Stereo Anywhere: Robust Zero-Shot Deep Stereo Matching Even Where Either Stereo or Mono Fail](https://mlsanity.com/paper/2412.04472/) (2024)
- [Revisiting Domain Generalized Stereo Matching Networks from a Feature Consistency Perspective](https://mlsanity.com/paper/2203.10887/) (2022)
- [NeRF-Supervised Deep Stereo](https://mlsanity.com/paper/2303.17603/) (2023)
- [GraftNet: Towards Domain Generalized Stereo Matching with a Broad-Spectrum and Task-Oriented Feature](https://mlsanity.com/paper/2204.00179/) (2022)
- [Bilevel Online Adaptation for Out-of-Domain Human Mesh Reconstruction](https://mlsanity.com/paper/2103.16449/) (2021)
- [Test-Time Training on Video Streams](https://mlsanity.com/paper/2307.05014/) (2023)
- [GIPSO: Geometrically Informed Propagation for Online Adaptation in 3D LiDAR Segmentation](https://mlsanity.com/paper/2207.09763/) (2022)
- [Booster: a Benchmark for Depth from Images of Specular and Transparent Surfaces](https://mlsanity.com/paper/2301.08245/) (2023)
- [Learning optical flow from still images](https://mlsanity.com/paper/2104.03965/) (2021)
- [LightStereo: Channel Boost Is All You Need for Efficient 2D Cost Aggregation](https://mlsanity.com/paper/2406.19833/) (2024)
- [A Decomposition Model for Stereo Matching](https://mlsanity.com/paper/2104.07516/) (2021)
- [Out-of-Domain Human Mesh Reconstruction via Dynamic Bilevel Online Adaptation](https://mlsanity.com/paper/2111.04017/) (2021)
- [On the confidence of stereo matching in a deep-learning era: a quantitative evaluation](https://mlsanity.com/paper/2101.00431/) (2021)
- [On the Road to Online Adaptation for Semantic Image Segmentation](https://mlsanity.com/paper/2203.16195/) (2022)
- [A Survey on Deep Stereo Matching in the Twenties](https://mlsanity.com/paper/2407.07816/) (2024)
- [Digging Into Uncertainty-based Pseudo-label for Robust Stereo Matching](https://mlsanity.com/paper/2307.16509/) (2023)
- [Neural Disparity Refinement for Arbitrary Resolution Stereo](https://mlsanity.com/paper/2110.15367/) (2021)
- [Online Domain Adaptation for Semantic Segmentation in Ever-Changing Conditions](https://mlsanity.com/paper/2207.10667/) (2022)
- [Improving Transformer-based Image Matching by Cascaded Capturing Spatially Informative Keypoints](https://mlsanity.com/paper/2303.02885/) (2023)
- [Robust Synthetic-to-Real Transfer for Stereo Matching](https://mlsanity.com/paper/2403.07705/) (2024)
- [Stereo Matching by Self-supervision of Multiscopic Vision](https://mlsanity.com/paper/2104.04170/) (2021)
- [To Adapt or Not to Adapt? Real-Time Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2307.15063/) (2023)
- [Federated Online Adaptation for Deep Stereo](https://mlsanity.com/paper/2405.14873/) (2024)
- [Estimating Image Depth in the Comics Domain](https://mlsanity.com/paper/2110.03575/) (2021)
- [Semantic Self-adaptation: Enhancing Generalization with a Single Sample](https://mlsanity.com/paper/2208.05788/) (2022)

## Abstract (arXiv)

Deep convolutional neural networks trained end-to-end are the state-of-the-art methods to regress dense disparity maps from stereo pairs. These models, however, suffer from a notable decrease in accuracy when exposed to scenarios significantly different from the training set, e.g., real vs synthetic images, etc.). We argue that it is extremely unlikely to gather enough samples to achieve effective training/tuning in any target domain, thus making this setup impractical for many applications. Instead, we propose to perform unsupervised and continuous online adaptation of a deep stereo network, which allows for preserving its accuracy in any environment. However, this strategy is extremely computationally demanding and thus prevents real-time inference. We address this issue introducing a new lightweight, yet effective, deep stereo architecture, Modularly ADaptive Network (MADNet) and developing a Modular ADaptation (MAD) algorithm, which independently trains sub-portions of the network. By deploying MADNet together with MAD we introduce the first real-time self-adaptive deep stereo system enabling competitive performance on heterogeneous datasets.