# HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening

arXiv: https://arxiv.org/abs/2203.02503 · HTML: https://mlsanity.com/paper/2203.02503/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 56 references that are in the map.
- First posted on arXiv 4 Mar 2022.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [A Transformer-Based Siamese Network for Change Detection](https://mlsanity.com/paper/2201.01293/) (2022)
- [Hyperspectral pansharpening: a review](https://mlsanity.com/paper/1504.04531/) (2015)
- [A convex formulation for hyperspectral image superresolution via subspace-based regularization](https://mlsanity.com/paper/1411.4005/) (2014)
- [Deep Gradient Projection Networks for Pan-sharpening](https://mlsanity.com/paper/2103.04584/) (2021)
- [Cross-Attention in Coupled Unmixing Nets for Unsupervised Hyperspectral Super-Resolution](https://mlsanity.com/paper/2007.05230/) (2020)
- [Hyperspectral and Multispectral Image Fusion based on a Sparse Representation](https://mlsanity.com/paper/1409.5729/) (2014)
- [Boosting the accuracy of multi-spectral image pan-sharpening by learning a deep residual network](https://mlsanity.com/paper/1705.07556/) (2017)
- [SPIN Road Mapper: Extracting Roads from Aerial Images via Spatial and Interaction Space Graph Reasoning for Autonomous Driving](https://mlsanity.com/paper/2109.07701/) (2021)
- [Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network](https://mlsanity.com/paper/1706.05249/) (2017)
- [Bayesian Fusion of Multi-Band Images](https://mlsanity.com/paper/1307.5996/) (2013)
- [Hyperspectral Pansharpening Based on Improved Deep Image Prior and Residual Reconstruction](https://mlsanity.com/paper/2107.02630/) (2021)
- [SIPSA-Net: Shift-Invariant Pan Sharpening with Moving Object Alignment for Satellite Imagery](https://mlsanity.com/paper/2105.02400/) (2021)

## Built on by (most cited first, 30 of 32)

- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [Changer: Feature Interaction is What You Need for Change Detection](https://mlsanity.com/paper/2209.08290/) (2022)
- [Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review](https://mlsanity.com/paper/2205.01380/) (2022)
- [Transformers in Remote Sensing: A Survey](https://mlsanity.com/paper/2209.01206/) (2022)
- [Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising](https://mlsanity.com/paper/2304.00844/) (2023)
- [Unsupervised Hyperspectral Pansharpening via Low-rank Diffusion Model](https://mlsanity.com/paper/2305.10925/) (2023)
- [ReFusion: Learning Image Fusion from Reconstruction with Learnable Loss via Meta-Learning](https://mlsanity.com/paper/2312.07943/) (2023)
- [2DQuant: Low-bit Post-Training Quantization for Image Super-Resolution](https://mlsanity.com/paper/2406.06649/) (2024)
- [GlobalMind: Global Multi-head Interactive Self-attention Network for Hyperspectral Change Detection](https://mlsanity.com/paper/2304.08687/) (2023)
- [A Spectral Diffusion Prior for Hyperspectral Image Super-Resolution](https://mlsanity.com/paper/2311.08955/) (2023)
- [Hyperspectral Pansharpening: Critical Review, Tools and Future Perspectives](https://mlsanity.com/paper/2407.01355/) (2024)
- [Multimodal Informative ViT: Information Aggregation and Distribution for Hyperspectral and LiDAR Classification](https://mlsanity.com/paper/2401.03179/) (2024)
- [Probability-based Global Cross-modal Upsampling for Pansharpening](https://mlsanity.com/paper/2303.13659/) (2023)
- [Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modeling](https://mlsanity.com/paper/2403.06978/) (2024)
- [CMT: Cross Modulation Transformer with Hybrid Loss for Pansharpening](https://mlsanity.com/paper/2404.01121/) (2024)
- [SRC-Net: Bi-Temporal Spatial Relationship Concerned Network for Change Detection](https://mlsanity.com/paper/2406.05668/) (2024)
- [When Images Look Right and Retrieve Wrong: Coverage-Guided Cross-Scale Re-Indexing for Knowledge-Faithful Generative Perception](https://mlsanity.com/paper/2608.20810/) (2026)
- [Exploiting Frequency Correlation for Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2406.00683/) (2024)
- [Multi-Head Attention Residual Unfolded Network for Model-Based Pansharpening](https://mlsanity.com/paper/2409.02675/) (2024)
- [Test-time Training for Hyperspectral Image Super-resolution](https://mlsanity.com/paper/2409.08667/) (2024)
- [Real HSI-MSI-PAN image dataset for the hyperspectral/multi-spectral/panchromatic image fusion and super-resolution fields](https://mlsanity.com/paper/2407.02387/) (2024)
- [A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusion](https://mlsanity.com/paper/2407.06159/) (2024)
- [HyFusion: Enhanced Reception Field Transformer for Hyperspectral Image Fusion](https://mlsanity.com/paper/2501.04665/) (2025)
- [Leveraging Large-Scale Pretrained Spatial-Spectral Priors for General Zero-Shot Pansharpening](https://mlsanity.com/paper/2512.02643/) (2025)
- [Model-Guided Network with Cluster-Based Operators for Spatio-Spectral Super-Resolution](https://mlsanity.com/paper/2505.24605/) (2025)
- [THAT: Token-wise High-frequency Augmentation Transformer for Hyperspectral Pansharpening](https://mlsanity.com/paper/2508.08183/) (2025)
- [Spatial-Spectral Binarized Neural Network for Panchromatic and Multi-spectral Images Fusion](https://mlsanity.com/paper/2509.23321/) (2025)
- [HSSDCT: Factorized Spatial-Spectral Correlation for Hyperspectral Image Fusion](https://mlsanity.com/paper/2602.00490/) (2026)
- [AMIF: Authorizable Medical Image Fusion Model with Built-in Authentication](https://mlsanity.com/paper/2603.24296/) (2026)
- [Euler-inspired Decoupling Neural Operator for Efficient Pansharpening](https://mlsanity.com/paper/2604.12463/) (2026)

## Abstract (arXiv)

Pansharpening aims to fuse a registered high-resolution panchromatic image (PAN) with a low-resolution hyperspectral image (LR-HSI) to generate an enhanced HSI with high spectral and spatial resolution. Existing pansharpening approaches neglect using an attention mechanism to transfer HR texture features from PAN to LR-HSI features, resulting in spatial and spectral distortions. In this paper, we present a novel attention mechanism for pansharpening called HyperTransformer, in which features of LR-HSI and PAN are formulated as queries and keys in a transformer, respectively. HyperTransformer consists of three main modules, namely two separate feature extractors for PAN and HSI, a multi-head feature soft attention module, and a spatial-spectral feature fusion module. Such a network improves both spatial and spectral quality measures of the pansharpened HSI by learning cross-feature space dependencies and long-range details of PAN and LR-HSI. Furthermore, HyperTransformer can be utilized across multiple spatial scales at the backbone for obtaining improved performance. Extensive experiments conducted on three widely used datasets demonstrate that HyperTransformer achieves significant improvement over the state-of-the-art methods on both spatial and spectral quality measures. Implementation code and pre-trained weights can be accessed at https://github.com/wgcban/HyperTransformer.