# Pre-Trained Image Processing Transformer

arXiv: https://arxiv.org/abs/2012.00364 · HTML: https://mlsanity.com/paper/2012.00364/

## Key facts

- Cited by 893 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2020.

## Built on by (most cited first, 30 of 893)

- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Simple Baselines for Image Restoration](https://mlsanity.com/paper/2204.04676/) (2022)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [Activating More Pixels in Image Super-Resolution Transformer](https://mlsanity.com/paper/2205.04437/) (2022)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [MambaIR: A Simple Baseline for Image Restoration with State-Space Model](https://mlsanity.com/paper/2402.15648/) (2024)
- [SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution](https://mlsanity.com/paper/2311.16518/) (2023)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [Remote Sensing Image Change Detection with Transformers](https://mlsanity.com/paper/2103.00208/) (2021)
- [MAXIM: Multi-Axis MLP for Image Processing](https://mlsanity.com/paper/2201.02973/) (2022)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)

## Abstract (arXiv)

As the computing power of modern hardware is increasing strongly, pre-trained deep learning models (e.g., BERT, GPT-3) learned on large-scale datasets have shown their effectiveness over conventional methods. The big progress is mainly contributed to the representation ability of transformer and its variant architectures. In this paper, we study the low-level computer vision task (e.g., denoising, super-resolution and deraining) and develop a new pre-trained model, namely, image processing transformer (IPT). To maximally excavate the capability of transformer, we present to utilize the well-known ImageNet benchmark for generating a large amount of corrupted image pairs. The IPT model is trained on these images with multi-heads and multi-tails. In addition, the contrastive learning is introduced for well adapting to different image processing tasks. The pre-trained model can therefore efficiently employed on desired task after fine-tuning. With only one pre-trained model, IPT outperforms the current state-of-the-art methods on various low-level benchmarks. Code is available at https://github.com/huawei-noah/Pretrained-IPT and https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/cv/IPT