# SiamTrans: Zero-Shot Multi-Frame Image Restoration with Pre-Trained Siamese Transformers

arXiv: https://arxiv.org/abs/2112.09426 · HTML: https://mlsanity.com/paper/2112.09426/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 68 references that are in the map.
- First posted on arXiv 17 Dec 2021.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Learning Continuous Image Representation with Local Implicit Image Function](https://mlsanity.com/paper/2012.09161/) (2020)
- [Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform](https://mlsanity.com/paper/1804.02815/) (2018)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [Multi-Scale Progressive Fusion Network for Single Image Deraining](https://mlsanity.com/paper/2003.10985/) (2020)
- [Density-aware Single Image De-raining using a Multi-stream Dense Network](https://mlsanity.com/paper/1802.07412/) (2018)
- [Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation](https://mlsanity.com/paper/2003.13659/) (2020)
- [Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining](https://mlsanity.com/paper/1807.05698/) (2018)
- [DesnowNet: Context-Aware Deep Network for Snow Removal](https://mlsanity.com/paper/1708.04512/) (2017)
- [Image Processing Using Multi-Code GAN Prior](https://mlsanity.com/paper/1912.07116/) (2019)
- [Semantic Photo Manipulation with a Generative Image Prior](https://mlsanity.com/paper/2005.07727/) (2020)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution](https://mlsanity.com/paper/2012.00739/) (2020)
- [Burst Denoising with Kernel Prediction Networks](https://mlsanity.com/paper/1712.02327/) (2017)
- [Attention-guided Network for Ghost-free High Dynamic Range Imaging](https://mlsanity.com/paper/1904.10293/) (2019)
- [Coherent Semantic Attention for Image Inpainting](https://mlsanity.com/paper/1905.12384/) (2019)
- [Handheld Multi-Frame Super-Resolution](https://mlsanity.com/paper/1905.03277/) (2019)
- [Video Super-resolution with Temporal Group Attention](https://mlsanity.com/paper/2007.10595/) (2020)
- [Wavelet-Based Dual-Branch Network for Image Demoireing](https://mlsanity.com/paper/2007.07173/) (2020)
- [Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework](https://mlsanity.com/paper/1803.10433/) (2018)
- [Deep Burst Denoising](https://mlsanity.com/paper/1712.05790/) (2017)
- [Image Demoireing with Learnable Bandpass Filters](https://mlsanity.com/paper/2004.00406/) (2020)

## Built on by (most cited first, 7 of 7)

- [TAPE: Task-Agnostic Prior Embedding for Image Restoration](https://mlsanity.com/paper/2203.06074/) (2022)
- [Low-Light Video Enhancement with Synthetic Event Guidance](https://mlsanity.com/paper/2208.11014/) (2022)
- [SiamixFormer: a fully-transformer Siamese network with temporal Fusion for accurate building detection and change detection in bi-temporal remote sensing images](https://mlsanity.com/paper/2208.00657/) (2022)
- [PAN-Crafter: Learning Modality-Consistent Alignment for PAN-Sharpening](https://mlsanity.com/paper/2505.23367/) (2025)
- [Efficient Dual-domain Image Dehazing with Haze Prior Perception](https://mlsanity.com/paper/2507.11035/) (2025)
- [Video Demoireing using Focused-Defocused Dual-Camera System](https://mlsanity.com/paper/2508.03449/) (2025)
- [DiffuMural: Restoring Dunhuang Murals with Multi-scale Diffusion](https://mlsanity.com/paper/2504.09513/) (2025)

## Abstract (arXiv)

We propose a novel zero-shot multi-frame image restoration method for removing unwanted obstruction elements (such as rains, snow, and moire patterns) that vary in successive frames. It has three stages: transformer pre-training, zero-shot restoration, and hard patch refinement. Using the pre-trained transformers, our model is able to tell the motion difference between the true image information and the obstructing elements. For zero-shot image restoration, we design a novel model, termed SiamTrans, which is constructed by Siamese transformers, encoders, and decoders. Each transformer has a temporal attention layer and several self-attention layers, to capture both temporal and spatial information of multiple frames. Only pre-trained (self-supervised) on the denoising task, SiamTrans is tested on three different low-level vision tasks (deraining, demoireing, and desnowing). Compared with related methods, ours achieves the best performances, even outperforming those with supervised learning.