# A self-supervised and adversarial approach to hyperspectral demosaicking and RGB reconstruction in surgical imaging

arXiv: https://arxiv.org/abs/2407.19282 · HTML: https://mlsanity.com/paper/2407.19282/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 53 references that are in the map.
- First posted on arXiv 27 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis](https://mlsanity.com/paper/1612.07919/) (2016)
- [Unsupervised Image Super-Resolution using Cycle-in-Cycle Generative Adversarial Networks](https://mlsanity.com/paper/1809.00437/) (2018)
- [NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results](https://mlsanity.com/paper/2005.01996/) (2020)
- [AdaptiveWeighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images](https://mlsanity.com/paper/2005.09305/) (2020)
- [Hyperspectral Image Segmentation: A Preliminary Study on the Oral and Dental Spectral Image Database (ODSI-DB)](https://mlsanity.com/paper/2303.08252/) (2023)
- [Adversarial training with cycle consistency for unsupervised super-resolution in endomicroscopy](https://mlsanity.com/paper/1901.06988/) (2019)
- [Deep Learning Approach for Hyperspectral Image Demosaicking, Spectral Correction and High-resolution RGB Reconstruction](https://mlsanity.com/paper/2109.01403/) (2021)
- [In-Vivo Hyperspectral Human Brain Image Database for Brain Cancer Detection](https://mlsanity.com/paper/2402.10776/) (2024)
- [The Shape of Sight: A Homological Framework for Unifying Visual Perception](https://mlsanity.com/paper/1802.04723/) (2018)
- [Unsupervised Spectral Demosaicing with Lightweight Spectral Attention Networks](https://mlsanity.com/paper/2307.01990/) (2023)
- [Spatial gradient consistency for unsupervised learning of hyperspectral demosaicking: Application to surgical imaging](https://mlsanity.com/paper/2302.10927/) (2023)

## Built on by (most cited first, 4 of 4)

- [Tree-based Semantic Losses: Application to Sparsely-supervised Large Multi-class Hyperspectral Segmentation](https://mlsanity.com/paper/2506.21150/) (2025)
- [X-RAFT: Cross-Modal Non-Rigid Registration of Blue and White Light Neurosurgical Hyperspectral Images](https://mlsanity.com/paper/2507.07747/) (2025)
- [Label tree semantic losses for rich multi-class medical image segmentation](https://mlsanity.com/paper/2507.15777/) (2025)
- [Perspective-Equivariant Fine-tuning for Multispectral Demosaicing without Ground Truth](https://mlsanity.com/paper/2603.01332/) (2026)

## Abstract (arXiv)

Hyperspectral imaging holds promises in surgical imaging by offering biological tissue differentiation capabilities with detailed information that is invisible to the naked eye. For intra-operative guidance, real-time spectral data capture and display is mandated. Snapshot mosaic hyperspectral cameras are currently seen as the most suitable technology given this requirement. However, snapshot mosaic imaging requires a demosaicking algorithm to fully restore the spatial and spectral details in the images. Modern demosaicking approaches typically rely on synthetic datasets to develop supervised learning methods, as it is practically impossible to simultaneously capture both snapshot and high-resolution spectral images of the exact same surgical scene. In this work, we present a self-supervised demosaicking and RGB reconstruction method that does not depend on paired high-resolution data as ground truth. We leverage unpaired standard high-resolution surgical microscopy images, which only provide RGB data but can be collected during routine surgeries. Adversarial learning complemented by self-supervised approaches are used to drive our hyperspectral-based RGB reconstruction into resembling surgical microscopy images and increasing the spatial resolution of our demosaicking. The spatial and spectral fidelity of the reconstructed hyperspectral images have been evaluated quantitatively. Moreover, a user study was conducted to evaluate the RGB visualisation generated from these spectral images. Both spatial detail and colour accuracy were assessed by neurosurgical experts. Our proposed self-supervised demosaicking method demonstrates improved results compared to existing methods, demonstrating its potential for seamless integration into intra-operative workflows.