# A Practical Stereo Depth System for Smart Glasses

arXiv: https://arxiv.org/abs/2211.10551 · HTML: https://mlsanity.com/paper/2211.10551/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 19 Nov 2022.

## Builds on (most cited first, 22 of 22)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [GA-Net: Guided Aggregation Net for End-to-end Stereo Matching](https://mlsanity.com/paper/1904.06587/) (2019)
- [Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation](https://mlsanity.com/paper/2203.11483/) (2022)
- [Hierarchical Neural Architecture Search for Deep Stereo Matching](https://mlsanity.com/paper/2010.13501/) (2020)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [Hierarchical Deep Stereo Matching on High-resolution Images](https://mlsanity.com/paper/1912.06704/) (2019)
- [HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching](https://mlsanity.com/paper/2007.12140/) (2020)
- [Hierarchical Discrete Distribution Decomposition for Match Density Estimation](https://mlsanity.com/paper/1812.06264/) (2018)
- [MobileStereoNet: Towards Lightweight Deep Networks for Stereo Matching](https://mlsanity.com/paper/2108.09770/) (2021)
- [Learning Stereo from Single Images](https://mlsanity.com/paper/2008.01484/) (2020)
- [One Shot 3D Photography](https://mlsanity.com/paper/2008.12298/) (2020)
- [Toward Practical Monocular Indoor Depth Estimation](https://mlsanity.com/paper/2112.02306/) (2021)
- [FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference](https://mlsanity.com/paper/2101.05615/) (2021)
- [Du2Net: Learning Depth Estimation from Dual-Cameras and Dual-Pixels](https://mlsanity.com/paper/2003.14299/) (2020)
- [Improving Deep Stereo Network Generalization with Geometric Priors](https://mlsanity.com/paper/2008.11098/) (2020)
- [High-Precision Online Markerless Stereo Extrinsic Calibration](https://mlsanity.com/paper/1903.10705/) (2019)

## Built on by (most cited first, 7 of 7)

- [Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Experts](https://mlsanity.com/paper/2507.04631/) (2025)
- [SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses](https://mlsanity.com/paper/2602.22683/) (2026)
- [Efficient Depth Estimation for Unstable Stereo Camera Systems on AR Glasses](https://mlsanity.com/paper/2411.10013/) (2024)
- [Color Mismatches in Stereoscopic Video: Real-World Dataset and Deep Correction Method](https://mlsanity.com/paper/2303.06657/) (2023)
- [SteROI-D: System Design and Mapping for Stereo Depth Inference on Regions of Interest](https://mlsanity.com/paper/2502.09528/) (2025)
- [CCNeXt: An Effective Self-Supervised Stereo Depth Estimation Approach](https://mlsanity.com/paper/2509.22627/) (2025)
- [MatchAttention: Embedding Explicit Matching Constraints into Attention for Efficient Stereo Matching](https://mlsanity.com/paper/2510.14260/) (2025)

## Abstract (arXiv)

We present the design of a productionized end-to-end stereo depth sensing system that does pre-processing, online stereo rectification, and stereo depth estimation with a fallback to monocular depth estimation when rectification is unreliable. The output of our depth sensing system is then used in a novel view generation pipeline to create 3D computational photography effects using point-of-view images captured by smart glasses. All these steps are executed on-device on the stringent compute budget of a mobile phone, and because we expect the users can use a wide range of smartphones, our design needs to be general and cannot be dependent on a particular hardware or ML accelerator such as a smartphone GPU. Although each of these steps is well studied, a description of a practical system is still lacking. For such a system, all these steps need to work in tandem with one another and fallback gracefully on failures within the system or less than ideal input data. We show how we handle unforeseen changes to calibration, e.g., due to heat, robustly support depth estimation in the wild, and still abide by the memory and latency constraints required for a smooth user experience. We show that our trained models are fast, and run in less than 1s on a six-year-old Samsung Galaxy S8 phone's CPU. Our models generalize well to unseen data and achieve good results on Middlebury and in-the-wild images captured from the smart glasses.