# Efficient and Explicit Modelling of Image Hierarchies for Image Restoration

arXiv: https://arxiv.org/abs/2303.00748 · HTML: https://mlsanity.com/paper/2303.00748/

## Key facts

- Cited by 145 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 97 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Builds on (most cited first, 30 of 66)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)

## Built on by (most cited first, 30 of 145)

- [MambaIR: A Simple Baseline for Image Restoration with State-Space Model](https://mlsanity.com/paper/2402.15648/) (2024)
- [AdaIR: Adaptive All-in-One Image Restoration via Frequency Mining and Modulation](https://mlsanity.com/paper/2403.14614/) (2024)
- [MambaIRv2: Attentive State Space Restoration](https://mlsanity.com/paper/2411.15269/) (2024)
- [Agent Attention: On the Integration of Softmax and Linear Attention](https://mlsanity.com/paper/2312.08874/) (2023)
- [HAT: Hybrid Attention Transformer for Image Restoration](https://mlsanity.com/paper/2309.05239/) (2023)
- [Transcending the Limit of Local Window: Advanced Super-Resolution Transformer with Adaptive Token Dictionary](https://mlsanity.com/paper/2401.08209/) (2024)
- [Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restoration](https://mlsanity.com/paper/2408.15994/) (2024)
- [A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends](https://mlsanity.com/paper/2410.15067/) (2024)
- [The Ninth NTIRE 2024 Efficient Super-Resolution Challenge Report](https://mlsanity.com/paper/2404.10343/) (2024)
- [LightM-UNet: Mamba Assists in Lightweight UNet for Medical Image Segmentation](https://mlsanity.com/paper/2403.05246/) (2024)
- [U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers](https://mlsanity.com/paper/2405.02730/) (2024)
- [Frequency-Assisted Mamba for Remote Sensing Image Super-Resolution](https://mlsanity.com/paper/2405.04964/) (2024)
- [Lightweight Adaptive Feature De-drifting for Compressed Image Classification](https://mlsanity.com/paper/2401.01724/) (2024)
- [NTIRE 2024 Challenge on Image Super-Resolution (x4): Methods and Results](https://mlsanity.com/paper/2404.09790/) (2024)
- [Efficient Real-world Image Super-Resolution Via Adaptive Directional Gradient Convolution](https://mlsanity.com/paper/2405.07023/) (2024)
- [Language-driven All-in-one Adverse Weather Removal](https://mlsanity.com/paper/2312.01381/) (2023)
- [Recursive Generalization Transformer for Image Super-Resolution](https://mlsanity.com/paper/2303.06373/) (2023)
- [Transfer CLIP for Generalizable Image Denoising](https://mlsanity.com/paper/2403.15132/) (2024)
- [Efficient Visual State Space Model for Image Deblurring](https://mlsanity.com/paper/2405.14343/) (2024)
- [Uncertainty-guided Perturbation for Image Super-Resolution Diffusion Model](https://mlsanity.com/paper/2503.18512/) (2025)
- [Progressive Focused Transformer for Single Image Super-Resolution](https://mlsanity.com/paper/2503.20337/) (2025)
- [Efficient Multi-scale Network with Learnable Discrete Wavelet Transform for Blind Motion Deblurring](https://mlsanity.com/paper/2401.00027/) (2023)
- [Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators](https://mlsanity.com/paper/2408.05710/) (2024)
- [From Zero to Detail: Deconstructing Ultra-High-Definition Image Restoration from Progressive Spectral Perspective](https://mlsanity.com/paper/2503.13165/) (2025)
- [Motion-adaptive Separable Collaborative Filters for Blind Motion Deblurring](https://mlsanity.com/paper/2404.13153/) (2024)
- [Efficient Face Super-Resolution via Wavelet-based Feature Enhancement Network](https://mlsanity.com/paper/2407.19768/) (2024)
- [LoRA-IR: Taming Low-Rank Experts for Efficient All-in-One Image Restoration](https://mlsanity.com/paper/2410.15385/) (2024)
- [Hybrid Agents for Image Restoration](https://mlsanity.com/paper/2503.10120/) (2025)
- [Hybrid Spectral Denoising Transformer with Guided Attention](https://mlsanity.com/paper/2303.09040/) (2023)
- [CascadedGaze: Efficiency in Global Context Extraction for Image Restoration](https://mlsanity.com/paper/2401.15235/) (2024)

## Abstract (arXiv)

The aim of this paper is to propose a mechanism to efficiently and explicitly model image hierarchies in the global, regional, and local range for image restoration. To achieve that, we start by analyzing two important properties of natural images including cross-scale similarity and anisotropic image features. Inspired by that, we propose the anchored stripe self-attention which achieves a good balance between the space and time complexity of self-attention and the modelling capacity beyond the regional range. Then we propose a new network architecture dubbed GRL to explicitly model image hierarchies in the Global, Regional, and Local range via anchored stripe self-attention, window self-attention, and channel attention enhanced convolution. Finally, the proposed network is applied to 7 image restoration types, covering both real and synthetic settings. The proposed method sets the new state-of-the-art for several of those. Code will be available at https://github.com/ofsoundof/GRL-Image-Restoration.git.