# A Learned Stereo Depth System for Robotic Manipulation in Homes

arXiv: https://arxiv.org/abs/2109.11644 · HTML: https://mlsanity.com/paper/2109.11644/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 23 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [AANet: Adaptive Aggregation Network for Efficient Stereo Matching](https://mlsanity.com/paper/2004.09548/) (2020)
- [Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras](https://mlsanity.com/paper/1904.04998/) (2019)
- [Understanding Convolution for Semantic Segmentation](https://mlsanity.com/paper/1702.08502/) (2017)
- [Hierarchical Neural Architecture Search for Deep Stereo Matching](https://mlsanity.com/paper/2010.13501/) (2020)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [Hierarchical Deep Stereo Matching on High-resolution Images](https://mlsanity.com/paper/1912.06704/) (2019)
- [HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching](https://mlsanity.com/paper/2007.12140/) (2020)
- [DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch](https://mlsanity.com/paper/1909.05845/) (2019)
- [StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction](https://mlsanity.com/paper/1807.08865/) (2018)
- [ActiveStereoNet: End-to-End Self-Supervised Learning for Active Stereo Systems](https://mlsanity.com/paper/1807.06009/) (2018)
- [AdaStereo: A Simple and Efficient Approach for Adaptive Stereo Matching](https://mlsanity.com/paper/2004.04627/) (2020)
- [On the Importance of Stereo for Accurate Depth Estimation: An Efficient Semi-Supervised Deep Neural Network Approach](https://mlsanity.com/paper/1803.09719/) (2018)
- [A Mobile Manipulation System for One-Shot Teaching of Complex Tasks in Homes](https://mlsanity.com/paper/1910.00127/) (2019)
- [Learning for Disparity Estimation through Feature Constancy](https://mlsanity.com/paper/1712.01039/) (2017)
- [Learning Stereo Matchability in Disparity Regression Networks](https://mlsanity.com/paper/2008.04800/) (2020)
- [Noise-Sampling Cross Entropy Loss: Improving Disparity Regression Via Cost Volume Aware Regularizer](https://mlsanity.com/paper/2005.08806/) (2020)

## Built on by (most cited first, 14 of 14)

- [CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects](https://mlsanity.com/paper/2303.15782/) (2023)
- [Guided Depth Map Super-resolution: A Survey](https://mlsanity.com/paper/2302.09598/) (2023)
- [ZeroGrasp: Zero-Shot Shape Reconstruction Enabled Robotic Grasping](https://mlsanity.com/paper/2504.10857/) (2025)
- [DIFT: Dynamic Iterative Field Transforms for Memory Efficient Optical Flow](https://mlsanity.com/paper/2306.05691/) (2023)
- [Revisiting Deformable Convolution for Depth Completion](https://mlsanity.com/paper/2308.01905/) (2023)
- [StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception](https://mlsanity.com/paper/2605.09989/) (2026)
- [Omni-Scan: Creating Visually-Accurate Digital Twin Object Models Using a Bimanual Robot with Handover and Gaussian Splat Merging](https://mlsanity.com/paper/2508.00354/) (2025)
- [SeeClear: Reliable Transparent Object Depth Estimation via Generative Opacification](https://mlsanity.com/paper/2603.19547/) (2026)
- [MaDis-Stereo: Enhanced Stereo Matching via Distilled Masked Image Modeling](https://mlsanity.com/paper/2409.02846/) (2024)
- [Simultaneous Pick and Place Detection by Combining SE(3) Diffusion Models with Differential Kinematics](https://mlsanity.com/paper/2504.19502/) (2025)
- [Sim-Grasp: Learning 6-DOF Grasp Policies for Cluttered Environments Using a Synthetic Benchmark](https://mlsanity.com/paper/2405.00841/) (2024)
- [A Direct Semi-Exhaustive Search Method for Robust, Partial-to-Full Point Cloud Registration](https://mlsanity.com/paper/2502.00115/) (2025)
- [GTR: Gaussian Splatting Tracking and Reconstruction of Unknown Objects Based on Appearance and Geometric Complexity](https://mlsanity.com/paper/2505.11905/) (2025)
- [Robust 6-DoF Object Pose Tracking with Built-In Recovery under Occlusions and Rapid Object Motions](https://mlsanity.com/paper/2607.23468/) (2026)

## Abstract (arXiv)

We present a passive stereo depth system that produces dense and accurate point clouds optimized for human environments, including dark, textureless, thin, reflective and specular surfaces and objects, at 2560x2048 resolution, with 384 disparities, in 30 ms. The system consists of an algorithm combining learned stereo matching with engineered filtering, a training and data-mixing methodology, and a sensor hardware design. Our architecture is 15x faster than approaches that perform similarly on the Middlebury and Flying Things Stereo Benchmarks. To effectively supervise the training of this model, we combine real data labelled using off-the-shelf depth sensors, as well as a number of different rendered, simulated labeled datasets. We demonstrate the efficacy of our system by presenting a large number of qualitative results in the form of depth maps and point-clouds, experiments validating the metric accuracy of our system and comparisons to other sensors on challenging objects and scenes. We also show the competitiveness of our algorithm compared to state-of-the-art learned models using the Middlebury and FlyingThings datasets.