# E2F-Net: Eyes-to-Face Inpainting via StyleGAN Latent Space

arXiv: https://arxiv.org/abs/2403.12197 · HTML: https://mlsanity.com/paper/2403.12197/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 54 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 30 of 30)

- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation](https://mlsanity.com/paper/2011.12799/) (2020)
- [HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing](https://mlsanity.com/paper/2111.15666/) (2021)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [Semantic Image Inpainting with Deep Generative Models](https://mlsanity.com/paper/1607.07539/) (2016)
- [Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks](https://mlsanity.com/paper/1711.06753/) (2017)
- [Coherent Semantic Attention for Image Inpainting](https://mlsanity.com/paper/1905.12384/) (2019)
- [Transforming and Projecting Images into Class-conditional Generative Networks](https://mlsanity.com/paper/2005.01703/) (2020)
- [WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose](https://mlsanity.com/paper/2005.10353/) (2020)
- [PEPSI++: Fast and Lightweight Network for Image Inpainting](https://mlsanity.com/paper/1905.09010/) (2019)
- [Texture Memory-Augmented Deep Patch-Based Image Inpainting](https://mlsanity.com/paper/2009.13240/) (2020)
- [Image Inpainting with External-internal Learning and Monochromic Bottleneck](https://mlsanity.com/paper/2104.09068/) (2021)
- [Progressive Image Inpainting with Full-Resolution Residual Network](https://mlsanity.com/paper/1907.10478/) (2019)
- [PiiGAN: Generative Adversarial Networks for Pluralistic Image Inpainting](https://mlsanity.com/paper/1912.01834/) (2019)
- [LaFIn: Generative Landmark Guided Face Inpainting](https://mlsanity.com/paper/1911.11394/) (2019)
- [FT-TDR: Frequency-guided Transformer and Top-Down Refinement Network for Blind Face Inpainting](https://mlsanity.com/paper/2108.04424/) (2021)
- [Facial Masks and Soft-Biometrics: Leveraging Face Recognition CNNs for Age and Gender Prediction on Mobile Ocular Images](https://mlsanity.com/paper/2103.16760/) (2021)
- [Repopulating Street Scenes](https://mlsanity.com/paper/2103.16183/) (2021)

## Built on by (most cited first, 1 of 1)

- [Face inpainting with Identity Preserving Latent Diffusion Models](https://mlsanity.com/paper/2605.16696/) (2026)

## Abstract (arXiv)

Face inpainting, the technique of restoring missing or damaged regions in facial images, is pivotal for applications like face recognition in occluded scenarios and image analysis with poor-quality captures. This process not only needs to produce realistic visuals but also preserve individual identity characteristics. The aim of this paper is to inpaint a face given periocular region (eyes-to-face) through a proposed new Generative Adversarial Network (GAN)-based model called Eyes-to-Face Network (E2F-Net). The proposed approach extracts identity and non-identity features from the periocular region using two dedicated encoders have been used. The extracted features are then mapped to the latent space of a pre-trained StyleGAN generator to benefit from its state-of-the-art performance and its rich, diverse and expressive latent space without any additional training. We further improve the StyleGAN output to find the optimal code in the latent space using a new optimization for GAN inversion technique. Our E2F-Net requires a minimum training process reducing the computational complexity as a secondary benefit. Through extensive experiments, we show that our method successfully reconstructs the whole face with high quality, surpassing current techniques, despite significantly less training and supervision efforts. We have generated seven eyes-to-face datasets based on well-known public face datasets for training and verifying our proposed methods. The code and datasets are publicly available.