# Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera

arXiv: https://arxiv.org/abs/2304.06019 · HTML: https://mlsanity.com/paper/2304.06019/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 66 references that are in the map.
- First posted on arXiv 12 Apr 2023.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [Toward Real-World Single Image Super-Resolution: A New Benchmark and A New Model](https://mlsanity.com/paper/1904.00523/) (2019)
- [DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks](https://mlsanity.com/paper/1711.07064/) (2017)
- [DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better](https://mlsanity.com/paper/1908.03826/) (2019)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [Learning a No-Reference Quality Metric for Single-Image Super-Resolution](https://mlsanity.com/paper/1612.05890/) (2016)
- [The Contextual Loss for Image Transformation with Non-Aligned Data](https://mlsanity.com/paper/1803.02077/) (2018)
- [Zoom To Learn, Learn To Zoom](https://mlsanity.com/paper/1905.05169/) (2019)

## Built on by (most cited first, 16 of 16)

- [Flare7K++: Mixing Synthetic and Real Datasets for Nighttime Flare Removal and Beyond](https://mlsanity.com/paper/2306.04236/) (2023)
- [Under-Display Camera Image Restoration with Scattering Effect](https://mlsanity.com/paper/2308.04163/) (2023)
- [Searching a Compact Architecture for Robust Multi-Exposure Image Fusion](https://mlsanity.com/paper/2305.12236/) (2023)
- [Flare-Aware Cross-modal Enhancement Network for Multi-spectral Vehicle Re-identification](https://mlsanity.com/paper/2305.13659/) (2023)
- [RGB Guided ToF Imaging System: A Survey of Deep Learning-based Methods](https://mlsanity.com/paper/2405.10357/) (2024)
- [Decoupling Degradations with Recurrent Network for Video Restoration in Under-Display Camera](https://mlsanity.com/paper/2403.05660/) (2024)
- [INRetouch: Context Aware Implicit Neural Representation for Photography Retouching](https://mlsanity.com/paper/2412.03848/) (2024)
- [Deep Video Restoration for Under-Display Camera](https://mlsanity.com/paper/2309.04752/) (2023)
- [DFDNet: Dynamic Frequency-Guided De-Flare Network](https://mlsanity.com/paper/2507.17489/) (2025)
- [Enhanced Super-Resolution Training via Mimicked Alignment for Real-World Scenes](https://mlsanity.com/paper/2410.05410/) (2024)
- [Integrating Spatial and Frequency Information for Under-Display Camera Image Restoration](https://mlsanity.com/paper/2501.18517/) (2025)
- [UDC-VIT: A Real-World Video Dataset for Under-Display Cameras](https://mlsanity.com/paper/2501.18545/) (2025)
- [Learnable Fractional Reaction-Diffusion Dynamics for Under-Display ToF Imaging and Beyond](https://mlsanity.com/paper/2511.01704/) (2025)
- [Segmentation Guided Sparse Transformer for Under-Display Camera Image Restoration](https://mlsanity.com/paper/2403.05906/) (2024)
- [Pseudo-Label Guided Real-World Image De-weathering: A Learning Framework with Imperfect Supervision](https://mlsanity.com/paper/2504.09949/) (2025)
- [UCMNet: Uncertainty-Aware Context Memory Network for Under-Display Camera Image Restoration](https://mlsanity.com/paper/2604.00381/) (2026)

## Abstract (arXiv)

Due to the difficulty in collecting large-scale and perfectly aligned paired training data for Under-Display Camera (UDC) image restoration, previous methods resort to monitor-based image systems or simulation-based methods, sacrificing the realness of the data and introducing domain gaps. In this work, we revisit the classic stereo setup for training data collection -- capturing two images of the same scene with one UDC and one standard camera. The key idea is to "copy" details from a high-quality reference image and "paste" them on the UDC image. While being able to generate real training pairs, this setting is susceptible to spatial misalignment due to perspective and depth of field changes. The problem is further compounded by the large domain discrepancy between the UDC and normal images, which is unique to UDC restoration. In this paper, we mitigate the non-trivial domain discrepancy and spatial misalignment through a novel Transformer-based framework that generates well-aligned yet high-quality target data for the corresponding UDC input. This is made possible through two carefully designed components, namely, the Domain Alignment Module (DAM) and Geometric Alignment Module (GAM), which encourage robust and accurate discovery of correspondence between the UDC and normal views. Extensive experiments show that high-quality and well-aligned pseudo UDC training pairs are beneficial for training a robust restoration network. Code and the dataset are available at https://github.com/jnjaby/AlignFormer.