# Dual Modality Prompted Diffusion Priors for Zero Shot Hyperspectral Pansharpening

arXiv: https://arxiv.org/abs/2608.11748 · HTML: https://mlsanity.com/paper/2608.11748/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 32 of its 57 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [IMAGDressing-v1: Customizable Virtual Dressing](https://mlsanity.com/paper/2407.12705/) (2024)
- [Hyperspectral pansharpening: a review](https://mlsanity.com/paper/1504.04531/) (2015)
- [A Multi-Scale and Multi-Depth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening](https://mlsanity.com/paper/1712.09809/) (2017)
- [Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model](https://mlsanity.com/paper/2502.09533/) (2025)
- [A convex formulation for hyperspectral image superresolution via subspace-based regularization](https://mlsanity.com/paper/1411.4005/) (2014)
- [MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classification](https://mlsanity.com/paper/2501.04944/) (2025)
- [HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models](https://mlsanity.com/paper/2402.15865/) (2024)
- [IMAGGarment: Fine-Grained Garment Generation for Controllable Fashion Design](https://mlsanity.com/paper/2504.13176/) (2025)
- [Unsupervised Sparse Dirichlet-Net for Hyperspectral Image Super-Resolution](https://mlsanity.com/paper/1804.05042/) (2018)
- [Boosting the accuracy of multi-spectral image pan-sharpening by learning a deep residual network](https://mlsanity.com/paper/1705.07556/) (2017)
- [Unsupervised Hyperspectral Pansharpening via Low-rank Diffusion Model](https://mlsanity.com/paper/2305.10925/) (2023)
- [Bayesian Fusion of Multi-Band Images](https://mlsanity.com/paper/1307.5996/) (2013)
- [Hyperspectral Pansharpening: Critical Review, Tools and Future Perspectives](https://mlsanity.com/paper/2407.01355/) (2024)
- [Hipandas: Hyperspectral Image Joint Denoising and Super-Resolution by Image Fusion with the Panchromatic Image](https://mlsanity.com/paper/2412.04201/) (2024)
- [Dynamic Cross-Modal Feature Interaction Network for Hyperspectral and LiDAR Data Classification](https://mlsanity.com/paper/2503.06945/) (2025)

## Abstract (arXiv)

Hyperspectral pansharpening aims to reconstruct a high resolution hyperspectral (HRHS) image from a panchromatic (PAN) image and a low resolution hyperspectral (LRHS) image while preserving both spatial details and spectral fidelity. Recent diffusion based methods exploit pretrained image priors by generating a low dimensional representation and subsequently mapping it to the hyperspectral domain. However, the observed panchromatic and hyperspectral images are typically imposed only through external reconstruction objectives, limiting their direct interaction with the diffusion prior. To address this issue, we propose dual-modality image-prompted diffusion model (DIDM) for zero shot hyperspectral pansharpening. DIDM encodes the low resolution hyperspectral and panchromatic observations into spectral and spatial prompt tokens, respectively, and injects them into intermediate features of a frozen remote sensing diffusion model through cross attention, allowing complementary spectral and spatial information to directly guide diffusion feature evolution. In addition, we introduce a panchromatic guided weighted pixel aware total variation regularizer that combines low resolution hyperspectral degradation fidelity and panchromatic response fidelity with gradient adaptive structural regularization, thereby preserving structural discontinuities while suppressing spurious variations in homogeneous regions. Extensive experiments on Pavia, Chikusei, and Houston under reduced resolution protocols show that DIDM achieves the best performance across all evaluated metrics, while full resolution evaluation on FR1 yields the highest HQNR among the compared methods. These results demonstrate that internal dual modality prompting and panchromatic guided structural regularization provide an effective balance between spatial detail enhancement and spectral preservation.