# 4D-MultispectralNet: Multispectral Stereoscopic Disparity Estimation using Human Masks

arXiv: https://arxiv.org/abs/2204.09089 · HTML: https://mlsanity.com/paper/2204.09089/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 20 references that are in the map.
- First posted on arXiv 19 Apr 2022.

## Builds on (most cited first, 8 of 8)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [Group-wise Correlation Stereo Network](https://mlsanity.com/paper/1903.04025/) (2019)
- [Computing the Stereo Matching Cost with a Convolutional Neural Network](https://mlsanity.com/paper/1409.4326/) (2014)
- [Online Mutual Foreground Segmentation for Multispectral Stereo Videos](https://mlsanity.com/paper/1809.02851/) (2018)
- [Domain Siamese CNNs for Sparse Multispectral Disparity Estimation](https://mlsanity.com/paper/2005.00088/) (2020)

## Built on by (most cited first, 1 of 1)

- [VisiTherS: Visible-thermal infrared stereo disparity estimation of human silhouette](https://mlsanity.com/paper/2304.11291/) (2023)

## Abstract (arXiv)

Multispectral stereoscopy is an emerging field. A lot of work has been done in classical stereoscopy, but multispectral stereoscopy is not studied as frequently. This type of stereoscopy can be used in autonomous vehicles to complete the information given by RGB cameras. It helps to identify objects in the surroundings when the conditions are more difficult, such as in night scenes. This paper focuses on the RGB-LWIR spectrum. RGB-LWIR stereoscopy has the same challenges as classical stereoscopy, that is occlusions, textureless surfaces and repetitive patterns, plus specific ones related to the different modalities. Finding matches between two spectrums adds another layer of complexity. Color, texture and shapes are more likely to vary from a spectrum to another. To address this additional challenge, this paper focuses on estimating the disparity of people present in a scene. Given the fact that people's shape is captured in both RGB and LWIR, we propose a novel method that uses segmentation masks of the human in both spectrum and than concatenate them to the original images before the first layer of a Siamese Network. This method helps to improve the accuracy, particularly within the one pixel error range.