# Learning GAN-based Foveated Reconstruction to Recover Perceptually Important Image Features

arXiv: https://arxiv.org/abs/2108.03499 · HTML: https://mlsanity.com/paper/2108.03499/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 71 references that are in the map.
- First posted on arXiv 7 Aug 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Generating Images with Perceptual Similarity Metrics based on Deep Networks](https://mlsanity.com/paper/1602.02644/) (2016)
- [Towards a quality metric for dense light fields](https://mlsanity.com/paper/1704.07576/) (2017)
- [Towards Metamerism via Foveated Style Transfer](https://mlsanity.com/paper/1705.10041/) (2017)
- [Central and peripheral vision for scene recognition: A neurocomputational modeling exploration](https://mlsanity.com/paper/1705.00816/) (2017)
- [Enforcing Perceptual Consistency on Generative Adversarial Networks by Using the Normalised Laplacian Pyramid Distance](https://mlsanity.com/paper/1908.04347/) (2019)
- [Perceptual Quality Assessment of Immersive Images Considering Peripheral Vision Impact](https://mlsanity.com/paper/1802.09065/) (2018)
- [Impacts of Retina-related Zones on Quality Perception of Omnidirectional Image](https://mlsanity.com/paper/1908.06239/) (2019)

## Built on by (most cited first, 1 of 1)

- [Gabor Fields: Orientation-Selective Level-of-Detail for Volume Rendering](https://mlsanity.com/paper/2602.05081/) (2026)

## Abstract (arXiv)

A foveated image can be entirely reconstructed from a sparse set of samples distributed according to the retinal sensitivity of the human visual system, which rapidly decreases with increasing eccentricity. The use of Generative Adversarial Networks has recently been shown to be a promising solution for such a task, as they can successfully hallucinate missing image information. As in the case of other supervised learning approaches, the definition of the loss function and the training strategy heavily influence the quality of the output. In this work,we consider the problem of efficiently guiding the training of foveated reconstruction techniques such that they are more aware of the capabilities and limitations of the human visual system, and thus can reconstruct visually important image features. Our primary goal is to make the training procedure less sensitive to distortions that humans cannot detect and focus on penalizing perceptually important artifacts. Given the nature of GAN-based solutions, we focus on the sensitivity of human vision to hallucination in case of input samples with different densities. We propose psychophysical experiments, a dataset, and a procedure for training foveated image reconstruction. The proposed strategy renders the generator network flexible by penalizing only perceptually important deviations in the output. As a result, the method emphasized the recovery of perceptually important image features. We evaluated our strategy and compared it with alternative solutions by using a newly trained objective metric, a recent foveated video quality metric, and user experiments. Our evaluations revealed significant improvements in the perceived image reconstruction quality compared with the standard GAN-based training approach.