# Stereo Matching by Self-supervision of Multiscopic Vision

arXiv: https://arxiv.org/abs/2104.04170 · HTML: https://mlsanity.com/paper/2104.04170/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 9 Apr 2021.

## Builds on (most cited first, 28 of 28)

- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [GA-Net: Guided Aggregation Net for End-to-end Stereo Matching](https://mlsanity.com/paper/1904.06587/) (2019)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [Real-time self-adaptive deep stereo](https://mlsanity.com/paper/1810.05424/) (2018)
- [Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1604.03650/) (2016)
- [Learning Monocular Depth by Distilling Cross-domain Stereo Networks](https://mlsanity.com/paper/1808.06586/) (2018)
- [Learning to Adapt for Stereo](https://mlsanity.com/paper/1904.02957/) (2019)
- [Parallax Attention for Unsupervised Stereo Correspondence Learning](https://mlsanity.com/paper/2009.08250/) (2020)
- [Flow2Stereo: Effective Self-Supervised Learning of Optical Flow and Stereo Matching](https://mlsanity.com/paper/2004.02138/) (2020)
- [Reversing the cycle: self-supervised deep stereo through enhanced monocular distillation](https://mlsanity.com/paper/2008.07130/) (2020)
- [Self-Supervised Learning for Stereo Matching with Self-Improving Ability](https://mlsanity.com/paper/1709.00930/) (2017)
- [Video Depth Estimation by Fusing Flow-to-Depth Proposals](https://mlsanity.com/paper/1912.12874/) (2019)
- [Open-World Stereo Video Matching with Deep RNN](https://mlsanity.com/paper/1808.03959/) (2018)
- [Depth Sensing Beyond LiDAR Range](https://mlsanity.com/paper/2004.03048/) (2020)
- [Reinforcement Learning in Topology-based Representation for Human Body Movement with Whole Arm Manipulation](https://mlsanity.com/paper/1809.04322/) (2018)
- [Active Perception with A Monocular Camera for Multiscopic Vision](https://mlsanity.com/paper/2001.08212/) (2020)

## Built on by (most cited first, 14 of 14)

- [NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation](https://mlsanity.com/paper/2203.01502/) (2022)
- [A Survey on Deep Stereo Matching in the Twenties](https://mlsanity.com/paper/2407.07816/) (2024)
- [Kick Back & Relax: Learning to Reconstruct the World by Watching SlowTV](https://mlsanity.com/paper/2307.10713/) (2023)
- [The Monocular Depth Estimation Challenge](https://mlsanity.com/paper/2211.12174/) (2022)
- [The Second Monocular Depth Estimation Challenge](https://mlsanity.com/paper/2304.07051/) (2023)
- [Multi-resolution Monocular Depth Map Fusion by Self-supervised Gradient-based Composition](https://mlsanity.com/paper/2212.01538/) (2022)
- [3D Former: Monocular Scene Reconstruction with 3D SDF Transformers](https://mlsanity.com/paper/2301.13510/) (2023)
- [Kick Back & Relax++: Scaling Beyond Ground-Truth Depth with SlowTV & CribsTV](https://mlsanity.com/paper/2403.01569/) (2024)
- [Self-Distilled Depth Refinement with Noisy Poisson Fusion](https://mlsanity.com/paper/2409.17880/) (2024)
- [A Portable Multiscopic Camera for Novel View and Time Synthesis in Dynamic Scenes](https://mlsanity.com/paper/2208.14433/) (2022)
- [Fine-tuning deep learning models for stereo matching using results from semi-global matching](https://mlsanity.com/paper/2205.14051/) (2022)
- [Pseudo-Stereo Inputs: A Solution to the Occlusion Challenge in Self-Supervised Stereo Matching](https://mlsanity.com/paper/2410.02534/) (2024)
- [MAGIC: Learning from Visibility Asymmetry for Unsupervised Stereo Matching](https://mlsanity.com/paper/2508.10838/) (2025)
- [BINO: Encoder Centric Self Supervised Stereo With Native Pair Input](https://mlsanity.com/paper/2603.27904/) (2026)

## Abstract (arXiv)

Self-supervised learning for depth estimation possesses several advantages over supervised learning. The benefits of no need for ground-truth depth, online fine-tuning, and better generalization with unlimited data attract researchers to seek self-supervised solutions. In this work, we propose a new self-supervised framework for stereo matching utilizing multiple images captured at aligned camera positions. A cross photometric loss, an uncertainty-aware mutual-supervision loss, and a new smoothness loss are introduced to optimize the network in learning disparity maps end-to-end without ground-truth depth information. To train this framework, we build a new multiscopic dataset consisting of synthetic images rendered by 3D engines and real images captured by real cameras. After being trained with only the synthetic images, our network can perform well in unseen outdoor scenes. Our experiment shows that our model obtains better disparity maps than previous unsupervised methods on the KITTI dataset and is comparable to supervised methods when generalized to unseen data. Our source code and dataset are available at https://sites.google.com/view/multiscopic.