# Open-World Stereo Video Matching with Deep RNN

arXiv: https://arxiv.org/abs/1808.03959 · HTML: https://mlsanity.com/paper/1808.03959/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2018.

## Built on by (most cited first, 20 of 20)

- [Implicit Motion Handling for Video Camouflaged Object Detection](https://mlsanity.com/paper/2203.07363/) (2022)
- [Deep Two-View Structure-from-Motion Revisited](https://mlsanity.com/paper/2104.00556/) (2021)
- [Test-Time Training on Video Streams](https://mlsanity.com/paper/2307.05014/) (2023)
- [Self-supervised Multi-view Stereo via Effective Co-Segmentation and Data-Augmentation](https://mlsanity.com/paper/2104.05374/) (2021)
- [Match-Stereo-Videos: Bidirectional Alignment for Consistent Dynamic Stereo Matching](https://mlsanity.com/paper/2403.10755/) (2024)
- [Vicinity Vision Transformer](https://mlsanity.com/paper/2206.10552/) (2022)
- [Generalizing to the Open World: Deep Visual Odometry with Online Adaptation](https://mlsanity.com/paper/2103.15279/) (2021)
- [Stereo Matching by Self-supervision of Multiscopic Vision](https://mlsanity.com/paper/2104.04170/) (2021)
- [Stereo Matching in Time: 100+ FPS Video Stereo Matching for Extended Reality](https://mlsanity.com/paper/2309.04183/) (2023)
- [TemporalStereo: Efficient Spatial-Temporal Stereo Matching Network](https://mlsanity.com/paper/2211.13755/) (2022)
- [Match Stereo Videos via Bidirectional Alignment](https://mlsanity.com/paper/2409.20283/) (2024)
- [Deep Laparoscopic Stereo Matching with Transformers](https://mlsanity.com/paper/2207.12152/) (2022)
- [Stereo Any Video: Temporally Consistent Stereo Matching](https://mlsanity.com/paper/2503.05549/) (2025)
- [On the Sins of Image Synthesis Loss for Self-supervised Depth Estimation](https://mlsanity.com/paper/2109.06163/) (2021)
- [Learning by Distillation: A Self-Supervised Learning Framework for Optical Flow Estimation](https://mlsanity.com/paper/2106.04195/) (2021)
- [Achieving Domain Robustness in Stereo Matching Networks by Removing Shortcut Learning](https://mlsanity.com/paper/2106.08486/) (2021)
- [Event Stream-based Visual Object Tracking: HDETrack V2 and A High-Definition Benchmark](https://mlsanity.com/paper/2502.05574/) (2025)
- [A High Resolution Multi-exposure Stereoscopic Image & Video Database of Natural Scenes](https://mlsanity.com/paper/2206.11095/) (2022)
- [MetaComp: Learning to Adapt for Online Depth Completion](https://mlsanity.com/paper/2207.10623/) (2022)
- [Forget, Anticipate and Adapt: Test Time Training for Long Videos](https://mlsanity.com/paper/2606.26515/) (2026)

## Abstract (arXiv)

Deep Learning based stereo matching methods have shown great successes and achieved top scores across different benchmarks. However, like most data-driven methods, existing deep stereo matching networks suffer from some well-known drawbacks such as requiring large amount of labeled training data, and that their performances are fundamentally limited by the generalization ability. In this paper, we propose a novel Recurrent Neural Network (RNN) that takes a continuous (possibly previously unseen) stereo video as input, and directly predicts a depth-map at each frame without a pre-training process, and without the need of ground-truth depth-maps as supervision. Thanks to the recurrent nature (provided by two convolutional-LSTM blocks), our network is able to memorize and learn from its past experiences, and modify its inner parameters (network weights) to adapt to previously unseen or unfamiliar environments. This suggests a remarkable generalization ability of the net, making it applicable in an em open world setting. Our method works robustly with changes in scene content, image statistics, and lighting and season conditions em etc. By extensive experiments, we demonstrate that the proposed method seamlessly adapts between different scenarios. Equally important, in terms of the stereo matching accuracy, it outperforms state-of-the-art deep stereo approaches on standard benchmark datasets such as KITTI and Middlebury stereo.