# Memory augment is All You Need for image restoration

arXiv: https://arxiv.org/abs/2309.01377 · HTML: https://mlsanity.com/paper/2309.01377/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 66 references that are in the map.
- First posted on arXiv 4 Sep 2023.

## Builds on (most cited first, 30 of 41)

- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Deep Joint Rain Detection and Removal from a Single Image](https://mlsanity.com/paper/1609.07769/) (2016)
- [Learning Enriched Features for Real Image Restoration and Enhancement](https://mlsanity.com/paper/2003.06792/) (2020)
- [MAXIM: Multi-Axis MLP for Image Processing](https://mlsanity.com/paper/2201.02973/) (2022)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Progressive Image Deraining Networks: A Better and Simpler Baseline](https://mlsanity.com/paper/1901.09221/) (2019)
- [Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation](https://mlsanity.com/paper/2003.13659/) (2020)
- [Contrastive Learning for Compact Single Image Dehazing](https://mlsanity.com/paper/2104.09367/) (2021)
- [CycleISP: Real Image Restoration via Improved Data Synthesis](https://mlsanity.com/paper/2003.07761/) (2020)
- [Attentive Generative Adversarial Network for Raindrop Removal from a Single Image](https://mlsanity.com/paper/1711.10098/) (2017)
- [Image De-raining Using a Conditional Generative Adversarial Network](https://mlsanity.com/paper/1701.05957/) (2017)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal](https://mlsanity.com/paper/1712.02478/) (2017)
- [Deep Generalized Unfolding Networks for Image Restoration](https://mlsanity.com/paper/2204.13348/) (2022)
- [Multi-Temporal Recurrent Neural Networks For Progressive Non-Uniform Single Image Deblurring With Incremental Temporal Training](https://mlsanity.com/paper/1911.07410/) (2019)
- [Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN](https://mlsanity.com/paper/1911.08718/) (2019)
- [Shadow Removal via Shadow Image Decomposition](https://mlsanity.com/paper/1908.08628/) (2019)
- [Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Data](https://mlsanity.com/paper/1903.10683/) (2019)
- [Auto-Exposure Fusion for Single-Image Shadow Removal](https://mlsanity.com/paper/2103.01255/) (2021)
- [DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network](https://mlsanity.com/paper/2207.10434/) (2022)
- [Direction-aware Spatial Context Features for Shadow Detection and Removal](https://mlsanity.com/paper/1805.04635/) (2018)
- [From Shadow Generation to Shadow Removal](https://mlsanity.com/paper/2103.12997/) (2021)
- [CANet: A Context-Aware Network for Shadow Removal](https://mlsanity.com/paper/2108.09894/) (2021)
- [Shadow Removal by a Lightness-Guided Network with Training on Unpaired Data](https://mlsanity.com/paper/2006.15617/) (2020)
- [RIS-GAN: Explore Residual and Illumination with Generative Adversarial Networks for Shadow Removal](https://mlsanity.com/paper/1911.09178/) (2019)
- [Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining](https://mlsanity.com/paper/1808.01491/) (2018)

## Built on by (most cited first, 8 of 8)

- [Diffusion Enhancement for Cloud Removal in Ultra-Resolution Remote Sensing Imagery](https://mlsanity.com/paper/2401.15105/) (2024)
- [High-Fidelity Document Stain Removal via A Large-Scale Real-World Dataset and A Memory-Augmented Transformer](https://mlsanity.com/paper/2410.22922/) (2024)
- [Effective Cloud Removal for Remote Sensing Images by an Improved Mean-Reverting Denoising Model with Elucidated Design Space](https://mlsanity.com/paper/2503.23717/) (2025)
- [Test-Time Intensity Consistency Adaptation for Shadow Detection](https://mlsanity.com/paper/2410.07695/) (2024)
- [DC4CR: When Cloud Removal Meets Diffusion Control in Remote Sensing](https://mlsanity.com/paper/2504.14785/) (2025)
- [Adaptive Query Prompting for Multi-Domain Landmark Detection](https://mlsanity.com/paper/2404.01194/) (2024)
- [Image Restoration via Multi-domain Learning](https://mlsanity.com/paper/2505.05504/) (2025)
- [ORBITALIF: An Efficient Spiking Federated Learning Framework for Onboard Cloud Removal](https://mlsanity.com/paper/2608.24073/) (2026)

## Abstract (arXiv)

Image restoration is a low-level vision task, most CNN methods are designed as a black box, lacking transparency and internal aesthetics. Although some methods combining traditional optimization algorithms with DNNs have been proposed, they all have some limitations. In this paper, we propose a three-granularity memory layer and contrast learning named MemoryNet, specifically, dividing the samples into positive, negative, and actual three samples for contrastive learning, where the memory layer is able to preserve the deep features of the image and the contrastive learning converges the learned features to balance. Experiments on Derain/Deshadow/Deblur task demonstrate that these methods are effective in improving restoration performance. In addition, this paper's model obtains significant PSNR, SSIM gain on three datasets with different degradation types, which is a strong proof that the recovered images are perceptually realistic. The source code of MemoryNet can be obtained from https://github.com/zhangbaijin/MemoryNet