# CMT: Cross Modulation Transformer with Hybrid Loss for Pansharpening

arXiv: https://arxiv.org/abs/2404.01121 · HTML: https://mlsanity.com/paper/2404.01121/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 38 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2111.07910/) (2021)
- [Pan-Mamba: Effective pan-sharpening with State Space Model](https://mlsanity.com/paper/2402.12192/) (2024)
- [HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening](https://mlsanity.com/paper/2203.02503/) (2022)
- [Fusformer: A Transformer-based Fusion Approach for Hyperspectral Image Super-resolution](https://mlsanity.com/paper/2109.02079/) (2021)
- [Pansharpening via Detail Injection Based Convolutional Neural Networks](https://mlsanity.com/paper/1806.08898/) (2018)
- [PanFormer: a Transformer Based Model for Pan-sharpening](https://mlsanity.com/paper/2203.02916/) (2022)
- [A Triple-Double Convolutional Neural Network for Panchromatic Sharpening](https://mlsanity.com/paper/2112.02237/) (2021)

## Built on by (most cited first, 4 of 4)

- [LightGen: Efficient Image Generation through Knowledge Distillation and Direct Preference Optimization](https://mlsanity.com/paper/2503.08619/) (2025)
- [Adaptive Rectangular Convolution for Remote Sensing Pansharpening](https://mlsanity.com/paper/2503.00467/) (2025)
- [Temporal Regularization Makes Your Video Generator Stronger](https://mlsanity.com/paper/2503.15417/) (2025)
- [DMAConv: Dual Mask-Adaptive Convolution for Remote Sensing Pansharpening](https://mlsanity.com/paper/2512.08331/) (2025)

## Abstract (arXiv)

Pansharpening aims to enhance remote sensing image (RSI) quality by merging high-resolution panchromatic (PAN) with multispectral (MS) images. However, prior techniques struggled to optimally fuse PAN and MS images for enhanced spatial and spectral information, due to a lack of a systematic framework capable of effectively coordinating their individual strengths. In response, we present the Cross Modulation Transformer (CMT), a pioneering method that modifies the attention mechanism. This approach utilizes a robust modulation technique from signal processing, integrating it into the attention mechanism's calculations. It dynamically tunes the weights of the carrier's value (V) matrix according to the modulator's features, thus resolving historical challenges and achieving a seamless integration of spatial and spectral attributes. Furthermore, considering that RSI exhibits large-scale features and edge details along with local textures, we crafted a hybrid loss function that combines Fourier and wavelet transforms to effectively capture these characteristics, thereby enhancing both spatial and spectral accuracy in pansharpening. Extensive experiments demonstrate our framework's superior performance over existing state-of-the-art methods. The code will be publicly available to encourage further research.