# Regularising disparity estimation via multi task learning with structured light reconstruction

arXiv: https://arxiv.org/abs/2301.08140 · HTML: https://mlsanity.com/paper/2301.08140/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 19 Jan 2023.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Stereo Correspondence and Reconstruction of Endoscopic Data Challenge](https://mlsanity.com/paper/2101.01133/) (2021)
- [Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks](https://mlsanity.com/paper/1605.02305/) (2016)
- [A Survey on Deep Learning Techniques for Stereo-based Depth Estimation](https://mlsanity.com/paper/2006.02535/) (2020)
- [Monocular Depth Estimation Based On Deep Learning: An Overview](https://mlsanity.com/paper/2003.06620/) (2020)
- [Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods](https://mlsanity.com/paper/1903.11260/) (2019)
- [A Close Look at Deep Learning with Small Data](https://mlsanity.com/paper/2003.12843/) (2020)

## Built on by (most cited first, 2 of 2)

- [SurgT challenge: Benchmark of Soft-Tissue Trackers for Robotic Surgery](https://mlsanity.com/paper/2302.03022/) (2023)
- [Dense 3D Reconstruction Through Lidar: A Comparative Study on Ex-vivo Porcine Tissue](https://mlsanity.com/paper/2401.10709/) (2024)

## Abstract (arXiv)

3D reconstruction is a useful tool for surgical planning and guidance. However, the lack of available medical data stunts research and development in this field, as supervised deep learning methods for accurate disparity estimation rely heavily on large datasets containing ground truth information. Alternative approaches to supervision have been explored, such as self-supervision, which can reduce or remove entirely the need for ground truth. However, no proposed alternatives have demonstrated performance capabilities close to what would be expected from a supervised setup. This work aims to alleviate this issue. In this paper, we investigate the learning of structured light projections to enhance the development of direct disparity estimation networks. We show for the first time that it is possible to accurately learn the projection of structured light on a scene, implicitly learning disparity. Secondly, we textcolorblackexplore the use of a multi task learning (MTL) framework for the joint training of structured light and disparity. We present results which show that MTL with structured light improves disparity training; without increasing the number of model parameters. Our MTL setup outperformed the single task learning (STL) network in every validation test. Notably, in the medical generalisation test, the STL error was 1.4 times worse than that of the best MTL performance. The benefit of using MTL is emphasised when the training data is limited. A dataset containing stereoscopic images, disparity maps and structured light projections on medical phantoms and ex vivo tissue was created for evaluation together with virtual scenes. This dataset will be made publicly available in the future.