# Deep Hyperspectral-Depth Reconstruction Using Single Color-Dot Projection

arXiv: https://arxiv.org/abs/2204.03929 · HTML: https://mlsanity.com/paper/2204.03929/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 8 Apr 2022.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics](https://mlsanity.com/paper/2009.00463/) (2020)
- [ActiveStereoNet: End-to-End Self-Supervised Learning for Active Stereo Systems](https://mlsanity.com/paper/1807.06009/) (2018)
- [AdaptiveWeighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images](https://mlsanity.com/paper/2005.09305/) (2020)
- [Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo](https://mlsanity.com/paper/2011.13117/) (2020)
- [Pro-Cam SSfM: Projector-Camera System for Structure and Spectral Reflectance from Motion](https://mlsanity.com/paper/1908.08185/) (2019)
- [Spectral MVIR: Joint Reconstruction of 3D Shape and Spectral Reflectance](https://mlsanity.com/paper/2104.07308/) (2021)
- [Hyperspectral Light Field Stereo Matching](https://mlsanity.com/paper/1709.00835/) (2017)
- [Spectral Reflectance Estimation Using Projector with Unknown Spectral Power Distribution](https://mlsanity.com/paper/2012.10083/) (2020)

## Built on by (most cited first, 4 of 4)

- [Dispersed Structured Light for Hyperspectral 3D Imaging](https://mlsanity.com/paper/2311.18287/) (2023)
- [LED: Light Enhanced Depth Estimation at Night](https://mlsanity.com/paper/2409.08031/) (2024)
- [Lumosaic: Hyperspectral Video via Active Illumination and Coded-Exposure Pixels](https://mlsanity.com/paper/2602.22140/) (2026)
- [Broadband Hyperspectral 3D Imaging using Dispersed Structured Light](https://mlsanity.com/paper/2605.25757/) (2026)

## Abstract (arXiv)

Depth reconstruction and hyperspectral reflectance reconstruction are two active research topics in computer vision and image processing. Conventionally, these two topics have been studied separately using independent imaging setups and there is no existing method which can acquire depth and spectral reflectance simultaneously in one shot without using special hardware. In this paper, we propose a novel single-shot hyperspectral-depth reconstruction method using an off-the-shelf RGB camera and projector. Our method is based on a single color-dot projection, which simultaneously acts as structured light for depth reconstruction and spatially-varying color illuminations for hyperspectral reflectance reconstruction. To jointly reconstruct the depth and the hyperspectral reflectance from a single color-dot image, we propose a novel end-to-end network architecture that effectively incorporates a geometric color-dot pattern loss and a photometric hyperspectral reflectance loss. Through the experiments, we demonstrate that our hyperspectral-depth reconstruction method outperforms the combination of an existing state-of-the-art single-shot hyperspectral reflectance reconstruction method and depth reconstruction method.