# MFuseNet: Robust Depth Estimation with Learned Multiscopic Fusion

arXiv: https://arxiv.org/abs/2108.02448 · HTML: https://mlsanity.com/paper/2108.02448/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 32 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 6 of 6)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [Building Generalizable Agents with a Realistic and Rich 3D Environment](https://mlsanity.com/paper/1801.02209/) (2018)
- [Tightly Coupled 3D Lidar Inertial Odometry and Mapping](https://mlsanity.com/paper/1904.06993/) (2019)
- [Reinforcement Learning in Topology-based Representation for Human Body Movement with Whole Arm Manipulation](https://mlsanity.com/paper/1809.04322/) (2018)

## Built on by (most cited first, 3 of 3)

- [NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation](https://mlsanity.com/paper/2203.01502/) (2022)
- [A Survey on RGB-D Datasets](https://mlsanity.com/paper/2201.05761/) (2022)
- [A Portable Multiscopic Camera for Novel View and Time Synthesis in Dynamic Scenes](https://mlsanity.com/paper/2208.14433/) (2022)

## Abstract (arXiv)

We design a multiscopic vision system that utilizes a low-cost monocular RGB camera to acquire accurate depth estimation. Unlike multi-view stereo with images captured at unconstrained camera poses, the proposed system controls the motion of a camera to capture a sequence of images in horizontally or vertically aligned positions with the same parallax. In this system, we propose a new heuristic method and a robust learning-based method to fuse multiple cost volumes between the reference image and its surrounding images. To obtain training data, we build a synthetic dataset with multiscopic images. The experiments on the real-world Middlebury dataset and real robot demonstration show that our multiscopic vision system outperforms traditional two-frame stereo matching methods in depth estimation. Our code and dataset are available at https://sites.google.com/view/multiscopic.