# Memory-Augmented Non-Local Attention for Video Super-Resolution

arXiv: https://arxiv.org/abs/2108.11048 · HTML: https://mlsanity.com/paper/2108.11048/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 25 Aug 2021.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution](https://mlsanity.com/paper/1704.03915/) (2017)
- [Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform](https://mlsanity.com/paper/1804.02815/) (2018)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [Deeply-Recursive Convolutional Network for Image Super-Resolution](https://mlsanity.com/paper/1511.04491/) (2015)
- [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://mlsanity.com/paper/2012.02181/) (2020)
- [Residual Non-local Attention Networks for Image Restoration](https://mlsanity.com/paper/1903.10082/) (2019)
- [TDAN: Temporally Deformable Alignment Network for Video Super-Resolution](https://mlsanity.com/paper/1812.02898/) (2018)
- [EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis](https://mlsanity.com/paper/1612.07919/) (2016)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [Frame-Recurrent Video Super-Resolution](https://mlsanity.com/paper/1801.04590/) (2018)
- [GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution](https://mlsanity.com/paper/2012.00739/) (2020)
- [Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation](https://mlsanity.com/paper/1611.05250/) (2016)
- [Detail-revealing Deep Video Super-resolution](https://mlsanity.com/paper/1704.02738/) (2017)
- [Memory-augmented Dense Predictive Coding for Video Representation Learning](https://mlsanity.com/paper/2008.01065/) (2020)

## Built on by (most cited first, 17 of 17)

- [MemFlow: Optical Flow Estimation and Prediction with Memory](https://mlsanity.com/paper/2404.04808/) (2024)
- [Swift Parameter-free Attention Network for Efficient Super-Resolution](https://mlsanity.com/paper/2311.12770/) (2023)
- [Learning Non-Local Spatial-Angular Correlation for Light Field Image Super-Resolution](https://mlsanity.com/paper/2302.08058/) (2023)
- [Benchmark Dataset and Effective Inter-Frame Alignment for Real-World Video Super-Resolution](https://mlsanity.com/paper/2212.05342/) (2022)
- [Local-Global Temporal Difference Learning for Satellite Video Super-Resolution](https://mlsanity.com/paper/2304.04421/) (2023)
- [Diverse 3D Hand Gesture Prediction from Body Dynamics by Bilateral Hand Disentanglement](https://mlsanity.com/paper/2303.01765/) (2023)
- [Structured Sparsity Learning for Efficient Video Super-Resolution](https://mlsanity.com/paper/2206.07687/) (2022)
- [Learning Truncated Causal History Model for Video Restoration](https://mlsanity.com/paper/2410.03936/) (2024)
- [TMP: Temporal Motion Propagation for Online Video Super-Resolution](https://mlsanity.com/paper/2312.09909/) (2023)
- [Driving-Video Dehazing with Non-Aligned Regularization for Safety Assistance](https://mlsanity.com/paper/2405.09996/) (2024)
- [Deep Blind Super-Resolution for Satellite Video](https://mlsanity.com/paper/2401.07139/) (2024)
- [A Survey of Deep Learning Video Super-Resolution](https://mlsanity.com/paper/2506.03216/) (2025)
- [SuperInpaint: Learning Detail-Enhanced Attentional Implicit Representation for Super-resolutional Image Inpainting](https://mlsanity.com/paper/2307.14489/) (2023)
- [Reference-based Restoration of Digitized Analog Videotapes](https://mlsanity.com/paper/2310.14926/) (2023)
- [Deep Learning Empowered Super-Resolution: A Comprehensive Survey and Future Prospects](https://mlsanity.com/paper/2509.22692/) (2025)
- [Towards Robust and Generalizable Continuous Space-Time Video Super-Resolution with Events](https://mlsanity.com/paper/2510.03833/) (2025)
- [ViStripformer: A Token-Efficient Transformer for Versatile Video Restoration](https://mlsanity.com/paper/2312.14502/) (2023)

## Abstract (arXiv)

In this paper, we propose a novel video super-resolution method that aims at generating high-fidelity high-resolution (HR) videos from low-resolution (LR) ones. Previous methods predominantly leverage temporal neighbor frames to assist the super-resolution of the current frame. Those methods achieve limited performance as they suffer from the challenge in spatial frame alignment and the lack of useful information from similar LR neighbor frames. In contrast, we devise a cross-frame non-local attention mechanism that allows video super-resolution without frame alignment, leading to be more robust to large motions in the video. In addition, to acquire the information beyond neighbor frames, we design a novel memory-augmented attention module to memorize general video details during the super-resolution training. Experimental results indicate that our method can achieve superior performance on large motion videos comparing to the state-of-the-art methods without aligning frames. Our source code will be released.