# Memory-augmented Deep Unfolding Network for Guided Image Super-resolution

arXiv: https://arxiv.org/abs/2203.04960 · HTML: https://mlsanity.com/paper/2203.04960/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 105 references that are in the map.
- First posted on arXiv 12 Feb 2022.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Accelerating the Super-Resolution Convolutional Neural Network](https://mlsanity.com/paper/1608.00367/) (2016)
- [Deeply-Recursive Convolutional Network for Image Super-Resolution](https://mlsanity.com/paper/1511.04491/) (2015)
- [Deep Unfolding Network for Image Super-Resolution](https://mlsanity.com/paper/2003.10428/) (2020)
- [Pixel-Adaptive Convolutional Neural Networks](https://mlsanity.com/paper/1904.05373/) (2019)
- [Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusion](https://mlsanity.com/paper/1910.04066/) (2019)
- [Deformable Kernel Networks for Joint Image Filtering](https://mlsanity.com/paper/1910.08373/) (2019)
- [A Multi-Scale and Multi-Depth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening](https://mlsanity.com/paper/1712.09809/) (2017)
- [Joint Image Filtering with Deep Convolutional Networks](https://mlsanity.com/paper/1710.04200/) (2017)
- [Fast End-to-End Trainable Guided Filter](https://mlsanity.com/paper/1803.05619/) (2018)
- [Multi-Contrast Super-Resolution MRI Through a Progressive Network](https://mlsanity.com/paper/1908.01612/) (2019)
- [Deep Gradient Projection Networks for Pan-sharpening](https://mlsanity.com/paper/2103.04584/) (2021)
- [Multi-Contrast MRI Super-Resolution via a Multi-Stage Integration Network](https://mlsanity.com/paper/2105.08949/) (2021)
- [Memory-Augmented Deep Unfolding Network for Compressive Sensing](https://mlsanity.com/paper/2110.09766/) (2021)
- [Super-Resolution with Deep Convolutional Sufficient Statistics](https://mlsanity.com/paper/1511.05666/) (2015)
- [Learning Scene Structure Guidance via Cross-Task Knowledge Transfer for Single Depth Super-Resolution](https://mlsanity.com/paper/2103.12955/) (2021)
- [Enhanced generative adversarial network for 3D brain MRI super-resolution](https://mlsanity.com/paper/1907.04835/) (2019)
- [Multimodal Deep Unfolding for Guided Image Super-Resolution](https://mlsanity.com/paper/2001.07575/) (2020)
- [Multimodal Image Super-resolution via Joint Sparse Representations induced by Coupled Dictionaries](https://mlsanity.com/paper/1709.08680/) (2017)
- [Multimodal Task-Driven Dictionary Learning for Image Classification](https://mlsanity.com/paper/1502.01094/) (2015)
- [Robust Guided Image Filtering](https://mlsanity.com/paper/1703.09379/) (2017)

## Built on by (most cited first, 25 of 25)

- [Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2303.08942/) (2023)
- [Deep Fourier Up-Sampling](https://mlsanity.com/paper/2210.05171/) (2022)
- [Guided Depth Map Super-resolution: A Survey](https://mlsanity.com/paper/2302.09598/) (2023)
- [Unaligned RGB Guided Hyperspectral Image Super-Resolution with Spatial-Spectral Concordance](https://mlsanity.com/paper/2505.02109/) (2025)
- [PanFlowNet: A Flow-Based Deep Network for Pan-sharpening](https://mlsanity.com/paper/2305.07774/) (2023)
- [SSDiff: Spatial-spectral Integrated Diffusion Model for Remote Sensing Pansharpening](https://mlsanity.com/paper/2404.11537/) (2024)
- [RPCANet++: Deep Interpretable Robust PCA for Sparse Object Segmentation](https://mlsanity.com/paper/2508.04190/) (2025)
- [Learning from Multi-Perception Features for Real-Word Image Super-resolution](https://mlsanity.com/paper/2305.18547/) (2023)
- [End-to-end Alternating Optimization for Real-World Blind Super Resolution](https://mlsanity.com/paper/2308.08816/) (2023)
- [RGB Guided ToF Imaging System: A Survey of Deep Learning-based Methods](https://mlsanity.com/paper/2405.10357/) (2024)
- [Deep Unfolding Multi-modal Image Fusion Network via Attribution Analysis](https://mlsanity.com/paper/2502.01467/) (2025)
- [Spiking Meets Attention: Efficient Remote Sensing Image Super-Resolution with Attention Spiking Neural Networks](https://mlsanity.com/paper/2503.04223/) (2025)
- [Learned Image Reasoning Prior Penetrates Deep Unfolding Network for Panchromatic and Multi-Spectral Image Fusion](https://mlsanity.com/paper/2308.16083/) (2023)
- [Scene Prior Filtering for Depth Super-Resolution](https://mlsanity.com/paper/2402.13876/) (2024)
- [Linearly-evolved Transformer for Pan-sharpening](https://mlsanity.com/paper/2404.12804/) (2024)
- [Probability-based Global Cross-modal Upsampling for Pansharpening](https://mlsanity.com/paper/2303.13659/) (2023)
- [Generalization analysis of an unfolding network for analysis-based Compressed Sensing](https://mlsanity.com/paper/2303.05582/) (2023)
- [PanBench: Towards High-Resolution and High-Performance Pansharpening](https://mlsanity.com/paper/2311.12083/) (2023)
- [MoCoLSK: Modality Conditioned High-Resolution Downscaling for Land Surface Temperature](https://mlsanity.com/paper/2409.19835/) (2024)
- [Structural Similarity-Inspired Unfolding for Lightweight Image Super-Resolution](https://mlsanity.com/paper/2506.11823/) (2025)
- [Multi-Order Matching Network for Alignment-Free Depth Super-Resolution](https://mlsanity.com/paper/2511.16361/) (2025)
- [Guided Image Restoration via Simultaneous Feature and Image Guided Fusion](https://mlsanity.com/paper/2312.08853/) (2023)
- [NTIRE 2024 Challenge on Stereo Image Super-Resolution: Methods and Results](https://mlsanity.com/paper/2409.16947/) (2024)
- [Fast Kernel-Space Diffusion for Remote Sensing Pansharpening](https://mlsanity.com/paper/2505.18991/) (2025)
- [Progressive Alignment Degradation Learning for Pansharpening](https://mlsanity.com/paper/2506.20179/) (2025)

## Abstract (arXiv)

Guided image super-resolution (GISR) aims to obtain a high-resolution (HR) target image by enhancing the spatial resolution of a low-resolution (LR) target image under the guidance of a HR image. However, previous model-based methods mainly takes the entire image as a whole, and assume the prior distribution between the HR target image and the HR guidance image, simply ignoring many non-local common characteristics between them. To alleviate this issue, we firstly propose a maximal a posterior (MAP) estimation model for GISR with two types of prior on the HR target image, i.e., local implicit prior and global implicit prior. The local implicit prior aims to model the complex relationship between the HR target image and the HR guidance image from a local perspective, and the global implicit prior considers the non-local auto-regression property between the two images from a global perspective. Secondly, we design a novel alternating optimization algorithm to solve this model for GISR. The algorithm is in a concise framework that facilitates to be replicated into commonly used deep network structures. Thirdly, to reduce the information loss across iterative stages, the persistent memory mechanism is introduced to augment the information representation by exploiting the Long short-term memory unit (LSTM) in the image and feature spaces. In this way, a deep network with certain interpretation and high representation ability is built. Extensive experimental results validate the superiority of our method on a variety of GISR tasks, including Pan-sharpening, depth image super-resolution, and MR image super-resolution.