# ISP-Agnostic Image Reconstruction for Under-Display Cameras

arXiv: https://arxiv.org/abs/2111.01511 · HTML: https://mlsanity.com/paper/2111.01511/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 2 Nov 2021.

## Builds on (most cited first, 17 of 17)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks](https://mlsanity.com/paper/1711.07064/) (2017)
- [DehazeNet: An End-to-End System for Single Image Haze Removal](https://mlsanity.com/paper/1601.07661/) (2016)
- [FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors](https://mlsanity.com/paper/1711.10703/) (2017)
- [Zoom To Learn, Learn To Zoom](https://mlsanity.com/paper/1905.05169/) (2019)
- [Replacing Mobile Camera ISP with a Single Deep Learning Model](https://mlsanity.com/paper/2002.05509/) (2020)
- [Gated Context Aggregation Network for Image Dehazing and Deraining](https://mlsanity.com/paper/1811.08747/) (2018)
- [Image Restoration for Under-Display Camera](https://mlsanity.com/paper/2003.04857/) (2020)
- [Towards Real Scene Super-Resolution with Raw Images](https://mlsanity.com/paper/1905.12156/) (2019)
- [Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report](https://mlsanity.com/paper/2105.07809/) (2021)
- [UDC 2020 Challenge on Image Restoration of Under-Display Camera: Methods and Results](https://mlsanity.com/paper/2008.07742/) (2020)
- [Transform Domain Pyramidal Dilated Convolution Networks For Restoration of Under Display Camera Images](https://mlsanity.com/paper/2009.09393/) (2020)
- [Deep Atrous Guided Filter for Image Restoration in Under Display Cameras](https://mlsanity.com/paper/2008.06229/) (2020)

## Built on by (most cited first, 3 of 3)

- [Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera](https://mlsanity.com/paper/2304.06019/) (2023)
- [Blind Face Restoration for Under-Display Camera via Dictionary Guided Transformer](https://mlsanity.com/paper/2308.10196/) (2023)
- [Segmentation Guided Sparse Transformer for Under-Display Camera Image Restoration](https://mlsanity.com/paper/2403.05906/) (2024)

## Abstract (arXiv)

Under-display cameras have been proposed in recent years as a way to reduce the form factor of mobile devices while maximizing the screen area. Unfortunately, placing the camera behind the screen results in significant image distortions, including loss of contrast, blur, noise, color shift, scattering artifacts, and reduced light sensitivity. In this paper, we propose an image-restoration pipeline that is ISP-agnostic, i.e. it can be combined with any legacy ISP to produce a final image that matches the appearance of regular cameras using the same ISP. This is achieved with a deep learning approach that performs a RAW-to-RAW image restoration. To obtain large quantities of real under-display camera training data with sufficient contrast and scene diversity, we furthermore develop a data capture method utilizing an HDR monitor, as well as a data augmentation method to generate suitable HDR content. The monitor data is supplemented with real-world data that has less scene diversity but allows us to achieve fine detail recovery without being limited by the monitor resolution. Together, this approach successfully restores color and contrast as well as image detail.