# Learning A 3D-CNN and Transformer Prior for Hyperspectral Image Super-Resolution

arXiv: https://arxiv.org/abs/2111.13923 · HTML: https://mlsanity.com/paper/2111.13923/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 73 references that are in the map.
- First posted on arXiv 27 Nov 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Bottleneck Transformers for Visual Recognition](https://mlsanity.com/paper/2101.11605/) (2021)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [A guide to convolution arithmetic for deep learning](https://mlsanity.com/paper/1603.07285/) (2016)
- [Non-Local Recurrent Network for Image Restoration](https://mlsanity.com/paper/1806.02919/) (2018)
- [Disentangled Non-Local Neural Networks](https://mlsanity.com/paper/2006.06668/) (2020)
- [Denoising Prior Driven Deep Neural Network for Image Restoration](https://mlsanity.com/paper/1801.06756/) (2018)
- [A convex formulation for hyperspectral image superresolution via subspace-based regularization](https://mlsanity.com/paper/1411.4005/) (2014)
- [Neural Proximal Gradient Descent for Compressive Imaging](https://mlsanity.com/paper/1806.03963/) (2018)
- [Medical Image Retrieval using Deep Convolutional Neural Network](https://mlsanity.com/paper/1703.08472/) (2017)
- [Fusformer: A Transformer-based Fusion Approach for Hyperspectral Image Super-resolution](https://mlsanity.com/paper/2109.02079/) (2021)
- [Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot Compressive Imaging](https://mlsanity.com/paper/2109.06548/) (2021)
- [NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image](https://mlsanity.com/paper/2005.03412/) (2020)
- [Advances in Hyperspectral Image Classification: Earth monitoring with statistical learning methods](https://mlsanity.com/paper/1310.5107/) (2013)
- [Hyperspectral and Multispectral Image Fusion based on a Sparse Representation](https://mlsanity.com/paper/1409.5729/) (2014)
- [Recent Advances and New Guidelines on Hyperspectral and Multispectral Image Fusion](https://mlsanity.com/paper/2008.03426/) (2020)
- [Hyperspectral Image Super-resolution via Deep Progressive Zero-centric Residual Learning](https://mlsanity.com/paper/2006.10300/) (2020)
- [Multi-Band Image Fusion Based on Spectral Unmixing](https://mlsanity.com/paper/1603.08720/) (2016)

## Built on by (most cited first, 10 of 10)

- [Unsupervised Hyperspectral Image Super-Resolution via Self-Supervised Modality Decoupling](https://mlsanity.com/paper/2412.04802/) (2024)
- [Minimalist and High-Quality Panoramic Imaging with PSF-aware Transformers](https://mlsanity.com/paper/2306.12992/) (2023)
- [A Novel State Space Model with Local Enhancement and State Sharing for Image Fusion](https://mlsanity.com/paper/2404.09293/) (2024)
- [Fourier-enhanced Implicit Neural Fusion Network for Multispectral and Hyperspectral Image Fusion](https://mlsanity.com/paper/2404.15174/) (2024)
- [Unsupervised Hyperspectral and Multispectral Image Blind Fusion Based on Deep Tucker Decomposition Network with Spatial-Spectral Manifold Learning](https://mlsanity.com/paper/2409.09670/) (2024)
- [Rethinking the Upsampling Layer in Hyperspectral Image Super Resolution](https://mlsanity.com/paper/2501.18664/) (2025)
- [Compact Multi-level-prior Tensor Representation for Hyperspectral Image Super-resolution](https://mlsanity.com/paper/2510.06098/) (2025)
- [A Cosine Network for Image Super-Resolution](https://mlsanity.com/paper/2601.16413/) (2026)
- [CoFusion: Multispectral and Hyperspectral Image Fusion via Spectral Coordinate Attention](https://mlsanity.com/paper/2604.10584/) (2026)
- [Adaptive 3D Convolution for Remote Sensing Image Fusion](https://mlsanity.com/paper/2605.09455/) (2026)

## Abstract (arXiv)

To solve the ill-posed problem of hyperspectral image super-resolution (HSISR), an usually method is to use the prior information of the hyperspectral images (HSIs) as a regularization term to constrain the objective function. Model-based methods using hand-crafted priors cannot fully characterize the properties of HSIs. Learning-based methods usually use a convolutional neural network (CNN) to learn the implicit priors of HSIs. However, the learning ability of CNN is limited, it only considers the spatial characteristics of the HSIs and ignores the spectral characteristics, and convolution is not effective for long-range dependency modeling. There is still a lot of room for improvement. In this paper, we propose a novel HSISR method that uses Transformer instead of CNN to learn the prior of HSIs. Specifically, we first use the proximal gradient algorithm to solve the HSISR model, and then use an unfolding network to simulate the iterative solution processes. The self-attention layer of Transformer makes it have the ability of spatial global interaction. In addition, we add 3D-CNN behind the Transformer layers to better explore the spatio-spectral correlation of HSIs. Both quantitative and visual results on two widely used HSI datasets and the real-world dataset demonstrate that the proposed method achieves a considerable gain compared to all the mainstream algorithms including the most competitive conventional methods and the recently proposed deep learning-based methods.