# Hyperspectral Image Super Resolution with Real Unaligned RGB Guidance

arXiv: https://arxiv.org/abs/2302.06298 · HTML: https://mlsanity.com/paper/2302.06298/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 60 references that are in the map.
- First posted on arXiv 13 Feb 2023.

## Builds on (most cited first, 17 of 17)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask](https://mlsanity.com/paper/2003.10955/) (2020)
- [CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping](https://mlsanity.com/paper/1807.10547/) (2018)
- [Learning to Synthesize a 4D RGBD Light Field from a Single Image](https://mlsanity.com/paper/1708.03292/) (2017)
- [Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery](https://mlsanity.com/paper/2005.08752/) (2020)
- [3D Quasi-Recurrent Neural Network for Hyperspectral Image Denoising](https://mlsanity.com/paper/2003.04547/) (2020)
- [Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net](https://mlsanity.com/paper/1901.03281/) (2019)
- [Weighted Low-rank Tensor Recovery for Hyperspectral Image Restoration](https://mlsanity.com/paper/1709.00192/) (2017)
- [Unsupervised Sparse Dirichlet-Net for Hyperspectral Image Super-Resolution](https://mlsanity.com/paper/1804.05042/) (2018)
- [Unsupervised and Unregistered Hyperspectral Image Super-Resolution with Mutual Dirichlet-Net](https://mlsanity.com/paper/1904.12175/) (2019)

## Built on by (most cited first, 13 of 13)

- [Frequency-aware Feature Fusion for Dense Image Prediction](https://mlsanity.com/paper/2408.12879/) (2024)
- [Frequency Dynamic Convolution for Dense Image Prediction](https://mlsanity.com/paper/2503.18783/) (2025)
- [Unaligned RGB Guided Hyperspectral Image Super-Resolution with Spatial-Spectral Concordance](https://mlsanity.com/paper/2505.02109/) (2025)
- [EventHDR: from Event to High-Speed HDR Videos and Beyond](https://mlsanity.com/paper/2409.17029/) (2024)
- [Quality Assessment for AI Generated Images with Instruction Tuning](https://mlsanity.com/paper/2405.07346/) (2024)
- [Siamese-DETR for Generic Multi-Object Tracking](https://mlsanity.com/paper/2310.17875/) (2023)
- [Enhancing Unregistered Hyperspectral Image Super-Resolution via Unmixing-based Abundance Fusion Learning](https://mlsanity.com/paper/2603.07918/) (2026)
- [HyperCap: Hyperspectral Land Cover Captioning Dataset for Vision Language Models](https://mlsanity.com/paper/2505.12217/) (2025)
- [Multispectral Demosaicing via Dual Cameras](https://mlsanity.com/paper/2503.22026/) (2025)
- [A Dual-Domain Convolutional Network for Hyperspectral Single-Image Super-Resolution](https://mlsanity.com/paper/2512.09546/) (2025)
- [Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observations](https://mlsanity.com/paper/2602.18822/) (2026)
- [Reinforcement Learning for Delivery Drone-Based Participatory Sensing in Dynamic Environments](https://mlsanity.com/paper/2607.18874/) (2026)
- [Lightweight Interpretable RGB-Guided Hyperspectral Super-Resolution under Real Cross-resolution Misalignment](https://mlsanity.com/paper/2609.01060/) (2026)

## Abstract (arXiv)

Fusion-based hyperspectral image (HSI) super-resolution has become increasingly prevalent for its capability to integrate high-frequency spatial information from the paired high-resolution (HR) RGB reference image. However, most of the existing methods either heavily rely on the accurate alignment between low-resolution (LR) HSIs and RGB images, or can only deal with simulated unaligned RGB images generated by rigid geometric transformations, which weakens their effectiveness for real scenes. In this paper, we explore the fusion-based HSI super-resolution with real RGB reference images that have both rigid and non-rigid misalignments. To properly address the limitations of existing methods for unaligned reference images, we propose an HSI fusion network with heterogenous feature extractions, multi-stage feature alignments, and attentive feature fusion. Specifically, our network first transforms the input HSI and RGB images into two sets of multi-scale features with an HSI encoder and an RGB encoder, respectively. The features of RGB reference images are then processed by a multi-stage alignment module to explicitly align the features of RGB reference with the LR HSI. Finally, the aligned features of RGB reference are further adjusted by an adaptive attention module to focus more on discriminative regions before sending them to the fusion decoder to generate the reconstructed HR HSI. Additionally, we collect a real-world HSI fusion dataset, consisting of paired HSI and unaligned RGB reference, to support the evaluation of the proposed model for real scenes. Extensive experiments are conducted on both simulated and our real-world datasets, and it shows that our method obtains a clear improvement over existing single-image and fusion-based super-resolution methods on quantitative assessment as well as visual comparison.