# Real-Time Semantic Stereo Matching

arXiv: https://arxiv.org/abs/1910.00541 · HTML: https://mlsanity.com/paper/1910.00541/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2019.

## Built on by (most cited first, 23 of 23)

- [SMD-Nets: Stereo Mixture Density Networks](https://mlsanity.com/paper/2104.03866/) (2021)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Open Challenges in Deep Stereo: the Booster Dataset](https://mlsanity.com/paper/2206.04671/) (2022)
- [Booster: a Benchmark for Depth from Images of Specular and Transparent Surfaces](https://mlsanity.com/paper/2301.08245/) (2023)
- [Self-supervised Multi-view Stereo via Effective Co-Segmentation and Data-Augmentation](https://mlsanity.com/paper/2104.05374/) (2021)
- [FastFlowNet: A Lightweight Network for Fast Optical Flow Estimation](https://mlsanity.com/paper/2103.04524/) (2021)
- [A Survey on Deep Stereo Matching in the Twenties](https://mlsanity.com/paper/2407.07816/) (2024)
- [Neural Disparity Refinement for Arbitrary Resolution Stereo](https://mlsanity.com/paper/2110.15367/) (2021)
- [Unknown Object Segmentation from Stereo Images](https://mlsanity.com/paper/2103.06796/) (2021)
- [S3M-Net: Joint Learning of Semantic Segmentation and Stereo Matching for Autonomous Driving](https://mlsanity.com/paper/2401.11414/) (2024)
- [RGB-Multispectral Matching: Dataset, Learning Methodology, Evaluation](https://mlsanity.com/paper/2206.07047/) (2022)
- [ORStereo: Occlusion-Aware Recurrent Stereo Matching for 4K-Resolution Images](https://mlsanity.com/paper/2103.07798/) (2021)
- [DCVSMNet: Double Cost Volume Stereo Matching Network](https://mlsanity.com/paper/2402.16473/) (2024)
- [TiCoSS: Tightening the Coupling between Semantic Segmentation and Stereo Matching within A Joint Learning Framework](https://mlsanity.com/paper/2407.18038/) (2024)
- [StereoVAE: A lightweight stereo-matching system using embedded GPUs](https://mlsanity.com/paper/2305.11566/) (2023)
- [Survey on Semantic Stereo Matching / Semantic Depth Estimation](https://mlsanity.com/paper/2109.10123/) (2021)
- [Efficient stereo matching on embedded GPUs with zero-means cross correlation](https://mlsanity.com/paper/2212.00476/) (2022)
- [Geometric Constraints in Deep Learning Frameworks: A Survey](https://mlsanity.com/paper/2403.12431/) (2024)
- [LiFR-Seg: Anytime High-Frame-Rate Segmentation via Event-Guided Propagation](https://mlsanity.com/paper/2603.21115/) (2026)
- [ESMStereo: Enhanced ShuffleMixer Disparity Upsampling for Real-Time and Accurate Stereo Matching](https://mlsanity.com/paper/2506.21091/) (2025)
- [Two-Stream Interactive Joint Learning of Scene Parsing and Geometric Vision Tasks](https://mlsanity.com/paper/2602.13588/) (2026)
- [Revisiting Matching Response and Swept Feature Volumes for Wide-baseline Omnidirectional Stereo](https://mlsanity.com/paper/2607.11097/) (2026)
- [StereoDiffuer: Diffusion-based Progressive Geometry Modeling with Saliency Attention Perception for Stereo Matching](https://mlsanity.com/paper/2608.21710/) (2026)

## Abstract (arXiv)

Scene understanding is paramount in robotics, self-navigation, augmented reality, and many other fields. To fully accomplish this task, an autonomous agent has to infer the 3D structure of the sensed scene (to know where it looks at) and its content (to know what it sees). To tackle the two tasks, deep neural networks trained to infer semantic segmentation and depth from stereo images are often the preferred choices. Specifically, Semantic Stereo Matching can be tackled by either standalone models trained for the two tasks independently or joint end-to-end architectures. Nonetheless, as proposed so far, both solutions are inefficient because requiring two forward passes in the former case or due to the complexity of a single network in the latter, although jointly tackling both tasks is usually beneficial in terms of accuracy. In this paper, we propose a single compact and lightweight architecture for real-time semantic stereo matching. Our framework relies on coarse-to-fine estimations in a multi-stage fashion, allowing: i) very fast inference even on embedded devices, with marginal drops in accuracy, compared to state-of-the-art networks, ii) trade accuracy for speed, according to the specific application requirements. Experimental results on high-end GPUs as well as on an embedded Jetson TX2 confirm the superiority of semantic stereo matching compared to standalone tasks and highlight the versatility of our framework on any hardware and for any application.