# Cross-modal Deep Face Normals with Deactivable Skip Connections

arXiv: https://arxiv.org/abs/2003.09691 · HTML: https://mlsanity.com/paper/2003.09691/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Mar 2020.

## Built on by (most cited first, 20 of 20)

- [Neural Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2112.01554/) (2021)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [Ensembling Off-the-shelf Models for GAN Training](https://mlsanity.com/paper/2112.09130/) (2021)
- [Facial Geometric Detail Recovery via Implicit Representation](https://mlsanity.com/paper/2203.09692/) (2022)
- [Video-driven Neural Physically-based Facial Asset for Production](https://mlsanity.com/paper/2202.05592/) (2022)
- [S2F2: Self-Supervised High Fidelity Face Reconstruction from Monocular Image](https://mlsanity.com/paper/2203.07732/) (2022)
- [Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection](https://mlsanity.com/paper/2106.07852/) (2021)
- [LumiGAN: Unconditional Generation of Relightable 3D Human Faces](https://mlsanity.com/paper/2304.13153/) (2023)
- [High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation](https://mlsanity.com/paper/2103.15876/) (2021)
- [SPARK: Self-supervised Personalized Real-time Monocular Face Capture](https://mlsanity.com/paper/2409.07984/) (2024)
- [Mesh-Tension Driven Expression-Based Wrinkles for Synthetic Faces](https://mlsanity.com/paper/2210.03529/) (2022)
- [Geometry-aware Two-scale PIFu Representation for Human Reconstruction](https://mlsanity.com/paper/2112.02082/) (2021)
- [AVFace: Towards Detailed Audio-Visual 4D Face Reconstruction](https://mlsanity.com/paper/2304.13115/) (2023)
- [Neural Relighting and Expression Transfer On Video Portraits](https://mlsanity.com/paper/2107.14735/) (2021)
- [Self-supervised Latent Space Optimization with Nebula Variational Coding](https://mlsanity.com/paper/2506.01414/) (2025)
- [GTAvatar: Bridging Gaussian Splatting and Texture Mapping for Relightable and Editable Gaussian Avatars](https://mlsanity.com/paper/2512.09162/) (2025)
- [Hybrid Approach for 3D Head Reconstruction: Using Neural Networks and Visual Geometry](https://mlsanity.com/paper/2104.13710/) (2021)
- [Monocular Human Shape and Pose with Dense Mesh-borne Local Image Features](https://mlsanity.com/paper/2111.05319/) (2021)
- [3D Face Reconstruction With Geometry Details From a Single Color Image Under Occluded Scenes](https://mlsanity.com/paper/2412.19849/) (2024)
- [Face Normal Estimation from Rags to Riches](https://mlsanity.com/paper/2601.01950/) (2026)

## Abstract (arXiv)

We present an approach for estimating surface normals from in-the-wild color images of faces. While data-driven strategies have been proposed for single face images, limited available ground truth data makes this problem difficult. To alleviate this issue, we propose a method that can leverage all available image and normal data, whether paired or not, thanks to a novel cross-modal learning architecture. In particular, we enable additional training with single modality data, either color or normal, by using two encoder-decoder networks with a shared latent space. The proposed architecture also enables face details to be transferred between the image and normal domains, given paired data, through skip connections between the image encoder and normal decoder. Core to our approach is a novel module that we call deactivable skip connections, which allows integrating both the auto-encoded and image-to-normal branches within the same architecture that can be trained end-to-end. This allows learning of a rich latent space that can accurately capture the normal information. We compare against state-of-the-art methods and show that our approach can achieve significant improvements, both quantitative and qualitative, with natural face images.