# Learning to Recover Spectral Reflectance from RGB Images

arXiv: https://arxiv.org/abs/2304.02162 · HTML: https://mlsanity.com/paper/2304.02162/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 84 references that are in the map.
- First posted on arXiv 4 Apr 2023.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Deep Learning for Image Super-resolution: A Survey](https://mlsanity.com/paper/1902.06068/) (2019)
- [Scale-recurrent Network for Deep Image Deblurring](https://mlsanity.com/paper/1802.01770/) (2018)
- [Meta-Transfer Learning for Few-Shot Learning](https://mlsanity.com/paper/1812.02391/) (2018)
- ["Zero-Shot" Super-Resolution using Deep Internal Learning](https://mlsanity.com/paper/1712.06087/) (2017)
- [Deep Learning for Hyperspectral Image Classification: An Overview](https://mlsanity.com/paper/1910.12861/) (2019)
- [Meta-Transfer Learning for Zero-Shot Super-Resolution](https://mlsanity.com/paper/2002.12213/) (2020)
- [Neural Blind Deconvolution Using Deep Priors](https://mlsanity.com/paper/1908.02197/) (2019)
- [Closed-loop Matters: Dual Regression Networks for Single Image Super-Resolution](https://mlsanity.com/paper/2003.07018/) (2020)
- [Self-Supervised Generalisation with Meta Auxiliary Learning](https://mlsanity.com/paper/1901.08933/) (2019)
- [MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction](https://mlsanity.com/paper/2204.07908/) (2022)
- [Deep Auxiliary Learning for Visual Localization and Odometry](https://mlsanity.com/paper/1803.03642/) (2018)
- [Fast Adaptation to Super-Resolution Networks via Meta-Learning](https://mlsanity.com/paper/2001.02905/) (2020)
- [Deep Hyperspectral Prior: Denoising, Inpainting, Super-Resolution](https://mlsanity.com/paper/1902.00301/) (2019)
- [Hierarchical Regression Network for Spectral Reconstruction from RGB Images](https://mlsanity.com/paper/2005.04703/) (2020)
- [Fine-Grained Visual Categorization using Meta-Learning Optimization with Sample Selection of Auxiliary Data](https://mlsanity.com/paper/1807.10916/) (2018)
- [Implicit Neural Representation Learning for Hyperspectral Image Super-Resolution](https://mlsanity.com/paper/2112.10541/) (2021)
- [Hyperspectral recovery from RGB images using Gaussian Processes](https://mlsanity.com/paper/1801.04654/) (2018)
- [Pixel-aware Deep Function-mixture Network for Spectral Super-Resolution](https://mlsanity.com/paper/1903.10501/) (2019)
- [Hyperspectral Image Super-resolution via Deep Progressive Zero-centric Residual Learning](https://mlsanity.com/paper/2006.10300/) (2020)
- [Adversarial Networks for Spatial Context-Aware Spectral Image Reconstruction from RGB](https://mlsanity.com/paper/1709.00265/) (2017)
- [Accurate Spectral Super-resolution from Single RGB Image Using Multi-scale CNN](https://mlsanity.com/paper/1806.03575/) (2018)
- [Blind Image Deconvolution Using Variational Deep Image Prior](https://mlsanity.com/paper/2202.00179/) (2022)
- [Tuning IR-cut Filter for Illumination-aware Spectral Reconstruction from RGB](https://mlsanity.com/paper/2103.14708/) (2021)

## Built on by (most cited first, 2 of 2)

- [Radiative-Structured Neural Operator for Continuous Spectral Super-Resolution](https://mlsanity.com/paper/2511.17895/) (2025)
- [USCTNet: A deep unfolding nuclear-norm optimization solver for physically consistent HSI reconstruction](https://mlsanity.com/paper/2509.10651/) (2025)

## Abstract (arXiv)

This paper tackles spectral reflectance recovery (SRR) from RGB images. Since capturing ground-truth spectral reflectance and camera spectral sensitivity are challenging and costly, most existing approaches are trained on synthetic images and utilize the same parameters for all unseen testing images, which are suboptimal especially when the trained models are tested on real images because they never exploit the internal information of the testing images. To address this issue, we adopt a self-supervised meta-auxiliary learning (MAXL) strategy that fine-tunes the well-trained network parameters with each testing image to combine external with internal information. To the best of our knowledge, this is the first work that successfully adapts the MAXL strategy to this problem. Instead of relying on naive end-to-end training, we also propose a novel architecture that integrates the physical relationship between the spectral reflectance and the corresponding RGB images into the network based on our mathematical analysis. Besides, since the spectral reflectance of a scene is independent to its illumination while the corresponding RGB images are not, we recover the spectral reflectance of a scene from its RGB images captured under multiple illuminations to further reduce the unknown. Qualitative and quantitative evaluations demonstrate the effectiveness of our proposed network and of the MAXL. Our code and data are available at https://github.com/Dong-Huo/SRR-MAXL.