# Limitations of Data-Driven Spectral Reconstruction -- An Optics-Aware Analysis

arXiv: https://arxiv.org/abs/2401.03835 · HTML: https://mlsanity.com/paper/2401.03835/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 111 references that are in the map.
- First posted on arXiv 8 Jan 2024.

## Builds on (most cited first, 26 of 26)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Learning Enriched Features for Real Image Restoration and Enhancement](https://mlsanity.com/paper/2003.06792/) (2020)
- [HINet: Half Instance Normalization Network for Image Restoration](https://mlsanity.com/paper/2105.06086/) (2021)
- [Data Augmentation Can Improve Robustness](https://mlsanity.com/paper/2111.05328/) (2021)
- [Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2111.07910/) (2021)
- [A Survey on Bias in Visual Datasets](https://mlsanity.com/paper/2107.07919/) (2021)
- [MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction](https://mlsanity.com/paper/2204.07908/) (2022)
- [HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging](https://mlsanity.com/paper/2203.02149/) (2022)
- [Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics](https://mlsanity.com/paper/2009.00463/) (2020)
- [Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net](https://mlsanity.com/paper/1901.03281/) (2019)
- [NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image](https://mlsanity.com/paper/2005.03412/) (2020)
- [Hierarchical Regression Network for Spectral Reconstruction from RGB Images](https://mlsanity.com/paper/2005.04703/) (2020)
- [AdaptiveWeighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images](https://mlsanity.com/paper/2005.09305/) (2020)
- [Binarized Spectral Compressive Imaging](https://mlsanity.com/paper/2305.10299/) (2023)
- [Recent Advances and New Guidelines on Hyperspectral and Multispectral Image Fusion](https://mlsanity.com/paper/2008.03426/) (2020)
- [A survey on computational spectral reconstruction methods from RGB to hyperspectral imaging](https://mlsanity.com/paper/2106.15944/) (2021)
- [Detecting Overfitting via Adversarial Examples](https://mlsanity.com/paper/1903.02380/) (2019)
- [Aberration-Aware Depth-from-Focus](https://mlsanity.com/paper/2303.04654/) (2023)
- [KRISM --- Krylov Subspace-based Optical Computing of Hyperspectral Images](https://mlsanity.com/paper/1801.09343/) (2018)
- [HPRN: Holistic Prior-embedded Relation Network for Spectral Super-Resolution](https://mlsanity.com/paper/2112.14608/) (2021)
- [One-to-Many Spectral Upsampling of Reflectances and Transmittances](https://mlsanity.com/paper/2306.11464/) (2023)
- [mHealth hyperspectral learning for instantaneous spatiospectral imaging of hemodynamics](https://mlsanity.com/paper/2303.16205/) (2023)

## Built on by (most cited first, 5 of 5)

- [Filterless Snapshot Hyperspectral Imaging using Guided Patch Diffusion](https://mlsanity.com/paper/2412.02798/) (2024)
- [Spectrum from Defocus: Fast Spectral Imaging with Chromatic Focal Stack](https://mlsanity.com/paper/2503.20184/) (2025)
- [Modulate and Reconstruct: Learning Hyperspectral Imaging from Misaligned Smartphone Views](https://mlsanity.com/paper/2507.01835/) (2025)
- [PHASE: Physiology-Aware Hyperspectral Reconstruction via Object-to-Human Domain Adaptation](https://mlsanity.com/paper/2511.13020/) (2025)
- [Uncertainty Quantification in HSI Reconstruction using Physics-Aware Diffusion Priors and Optics-Encoded Measurements](https://mlsanity.com/paper/2511.18473/) (2025)

## Abstract (arXiv)

Hyperspectral imaging empowers machine vision systems with the distinct capability of identifying materials through recording their spectral signatures. Recent efforts in data-driven spectral reconstruction aim at extracting spectral information from RGB images captured by cost-effective RGB cameras, instead of dedicated hardware. Published work reports exceedingly high numerical scores for this reconstruction task, yet real-world performance lags substantially behind. We systematically analyze the performance of such methods. First, we evaluate the overfitting limitations with respect to current datasets by training the networks with less data, validating the trained models with unseen yet slightly modified data and cross-dataset validation. Second, we reveal fundamental limitations in the ability of RGB to spectral methods to deal with metameric or near-metameric conditions, which have so far gone largely unnoticed due to the insufficiencies of existing datasets. We validate the trained models with metamer data generated by metameric black theory and re-training the networks with various forms of metamers. This methodology can also be used for data augmentation as a partial mitigation of the dataset issues, although the RGB to spectral inverse problem remains fundamentally ill-posed. Finally, we analyze the potential for modifying the problem setting to achieve better performance by exploiting optical encoding provided by either optical aberrations or deliberate optical design. Our experiments show such approaches provide improved results under certain circumstances, but their overall performance is limited by the same dataset issues. We conclude that future progress on snapshot spectral imaging will heavily depend on the generation of improved datasets which can then be used to design effective optical encoding strategies. Code: https://github.com/vccimaging/OpticsAwareHSI-Analysis.