# HAT: Hybrid Attention Transformer for Image Restoration

arXiv: https://arxiv.org/abs/2309.05239 · HTML: https://mlsanity.com/paper/2309.05239/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 81 of its 116 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 30 of 81)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data](https://mlsanity.com/paper/2107.10833/) (2021)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)

## Built on by (most cited first, 30 of 66)

- [A Comparative Study of Image Restoration Networks for General Backbone Network Design](https://mlsanity.com/paper/2310.11881/) (2023)
- [U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers](https://mlsanity.com/paper/2405.02730/) (2024)
- [Towards Effective Multiple-in-One Image Restoration: A Sequential and Prompt Learning Strategy](https://mlsanity.com/paper/2401.03379/) (2024)
- [Pixel to Gaussian: Ultra-Fast Continuous Super-Resolution with 2D Gaussian Modeling](https://mlsanity.com/paper/2503.06617/) (2025)
- [Neural Degradation Representation Learning for All-In-One Image Restoration](https://mlsanity.com/paper/2310.12848/) (2023)
- [SRGS: Super-Resolution 3D Gaussian Splatting](https://mlsanity.com/paper/2404.10318/) (2024)
- [DVMSR: Distillated Vision Mamba for Efficient Super-Resolution](https://mlsanity.com/paper/2405.03008/) (2024)
- [SuperGaussian: Repurposing Video Models for 3D Super Resolution](https://mlsanity.com/paper/2406.00609/) (2024)
- [Learning A Low-Level Vision Generalist via Visual Task Prompt](https://mlsanity.com/paper/2408.08601/) (2024)
- [MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution](https://mlsanity.com/paper/2404.09571/) (2024)
- [MoE-DiffIR: Task-customized Diffusion Priors for Universal Compressed Image Restoration](https://mlsanity.com/paper/2407.10833/) (2024)
- [WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learning](https://mlsanity.com/paper/2411.05420/) (2024)
- [The Fourth Challenge on Image Super-Resolution (×4) at NTIRE 2026: Benchmark Results and Method Overview](https://mlsanity.com/paper/2604.14558/) (2026)
- [NTIRE 2025 Challenge on RAW Image Restoration and Super-Resolution](https://mlsanity.com/paper/2506.02197/) (2025)
- [Exploiting Diffusion Prior for Task-driven Image Restoration](https://mlsanity.com/paper/2507.22459/) (2025)
- [The First Challenge on Remote Sensing Infrared Image Super-Resolution at NTIRE 2026: Benchmark Results and Method Overview](https://mlsanity.com/paper/2604.21312/) (2026)
- [Deep RAW Image Super-Resolution. A NTIRE 2024 Challenge Survey](https://mlsanity.com/paper/2404.16223/) (2024)
- [Geometric Distortion Guided Transformer for Omnidirectional Image Super-Resolution](https://mlsanity.com/paper/2406.10869/) (2024)
- [DiffSR: Learning Radar Reflectivity Synthesis via Diffusion Model from Satellite Observations](https://mlsanity.com/paper/2411.06714/) (2024)
- [PBR-SR: Mesh PBR Texture Super Resolution from 2D Image Priors](https://mlsanity.com/paper/2506.02846/) (2025)
- [ICPR 2026 Competition on Low-Resolution License Plate Recognition](https://mlsanity.com/paper/2604.22506/) (2026)
- [Self-Evolving Agentic Image Restoration via Deliberate Planning and Intuitive Execution](https://mlsanity.com/paper/2606.28971/) (2026)
- [Efficient Image Deblurring Networks based on Diffusion Models](https://mlsanity.com/paper/2401.05907/) (2024)
- [ControlSR: Taming Diffusion Models for Consistent Real-World Image Super Resolution](https://mlsanity.com/paper/2410.14279/) (2024)
- [SR3R: Rethinking Super-Resolution 3D Reconstruction With Feed-Forward Gaussian Splatting](https://mlsanity.com/paper/2602.24020/) (2026)
- [PupilSense: A Novel Application for Webcam-Based Pupil Diameter Estimation](https://mlsanity.com/paper/2407.11204/) (2024)
- [MatIR: A Hybrid Mamba-Transformer Image Restoration Model](https://mlsanity.com/paper/2501.18401/) (2025)
- [PanoDiff-SR: Synthesizing Dental Panoramic Radiographs using Diffusion and Super-resolution](https://mlsanity.com/paper/2507.09227/) (2025)
- [MF-LPR2: Multi-Frame License Plate Image Restoration and Recognition using Optical Flow](https://mlsanity.com/paper/2508.14797/) (2025)
- [ShiftLUT: Spatial Shift Enhanced Look-Up Tables for Efficient Image Restoration](https://mlsanity.com/paper/2603.00906/) (2026)

## Abstract (arXiv)

Transformer-based methods have shown impressive performance in image restoration tasks, such as image super-resolution and denoising. However, we find that these networks can only utilize a limited spatial range of input information through attribution analysis. This implies that the potential of Transformer is still not fully exploited in existing networks. In order to activate more input pixels for better restoration, we propose a new Hybrid Attention Transformer (HAT). It combines both channel attention and window-based self-attention schemes, thus making use of their complementary advantages. Moreover, to better aggregate the cross-window information, we introduce an overlapping cross-attention module to enhance the interaction between neighboring window features. In the training stage, we additionally adopt a same-task pre-training strategy to further exploit the potential of the model for further improvement. Extensive experiments have demonstrated the effectiveness of the proposed modules. We further scale up the model to show that the performance of the SR task can be greatly improved. Besides, we extend HAT to more image restoration applications, including real-world image super-resolution, Gaussian image denoising and image compression artifacts reduction. Experiments on benchmark and real-world datasets demonstrate that our HAT achieves state-of-the-art performance both quantitatively and qualitatively. Codes and models are publicly available at https://github.com/XPixelGroup/HAT.