# Physically-Based Face Rendering for NIR-VIS Face Recognition

arXiv: https://arxiv.org/abs/2211.06408 · HTML: https://mlsanity.com/paper/2211.06408/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 66 references that are in the map.
- First posted on arXiv 11 Nov 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2103.04098/) (2021)
- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [A Light CNN for Deep Face Representation with Noisy Labels](https://mlsanity.com/paper/1511.02683/) (2015)
- [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017)
- [AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"](https://mlsanity.com/paper/2003.13845/) (2020)
- [Sample and Computation Redistribution for Efficient Face Detection](https://mlsanity.com/paper/2105.04714/) (2021)
- [UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognition](https://mlsanity.com/paper/1712.04695/) (2017)
- [Domain Adaptive Neural Networks for Object Recognition](https://mlsanity.com/paper/1409.6041/) (2014)
- [AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs](https://mlsanity.com/paper/2112.05957/) (2021)
- [Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing](https://mlsanity.com/paper/2103.15432/) (2021)
- [WebFace260M: A Benchmark for Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2204.10149/) (2022)
- [Wasserstein CNN: Learning Invariant Features for NIR-VIS Face Recognition](https://mlsanity.com/paper/1708.02412/) (2017)
- [OSTeC: One-Shot Texture Completion](https://mlsanity.com/paper/2012.15370/) (2020)
- [Normalized Avatar Synthesis Using StyleGAN and Perceptual Refinement](https://mlsanity.com/paper/2106.11423/) (2021)
- [Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networks](https://mlsanity.com/paper/1909.02215/) (2019)
- [DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition](https://mlsanity.com/paper/2009.09399/) (2020)
- [Dual Variational Generation for Low-Shot Heterogeneous Face Recognition](https://mlsanity.com/paper/1903.10203/) (2019)
- [Synthesis of High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks](https://mlsanity.com/paper/1812.05155/) (2018)
- [Not Afraid of the Dark: NIR-VIS Face Recognition via Cross-spectral Hallucination and Low-rank Embedding](https://mlsanity.com/paper/1611.06638/) (2016)
- [Disentangled Variational Representation for Heterogeneous Face Recognition](https://mlsanity.com/paper/1809.01936/) (2018)
- [Cross-Spectral Face Hallucination via Disentangling Independent Factors](https://mlsanity.com/paper/1909.04365/) (2019)
- [Adversarial Discriminative Heterogeneous Face Recognition](https://mlsanity.com/paper/1709.03675/) (2017)
- [LAMP-HQ: A Large-Scale Multi-Pose High-Quality Database and Benchmark for NIR-VIS Face Recognition](https://mlsanity.com/paper/1912.07809/) (2019)

## Built on by (most cited first, 3 of 3)

- [FitMe: Deep Photorealistic 3D Morphable Model Avatars](https://mlsanity.com/paper/2305.09641/) (2023)
- [Rethinking the Domain Gap in Near-infrared Face Recognition](https://mlsanity.com/paper/2312.00627/) (2023)
- [Human-Imperceptible Physical Adversarial Attack for NIR Face Recognition Models](https://mlsanity.com/paper/2504.15823/) (2025)

## Abstract (arXiv)

Near infrared (NIR) to Visible (VIS) face matching is challenging due to the significant domain gaps as well as a lack of sufficient data for cross-modality model training. To overcome this problem, we propose a novel method for paired NIR-VIS facial image generation. Specifically, we reconstruct 3D face shape and reflectance from a large 2D facial dataset and introduce a novel method of transforming the VIS reflectance to NIR reflectance. We then use a physically-based renderer to generate a vast, high-resolution and photorealistic dataset consisting of various poses and identities in the NIR and VIS spectra. Moreover, to facilitate the identity feature learning, we propose an IDentity-based Maximum Mean Discrepancy (ID-MMD) loss, which not only reduces the modality gap between NIR and VIS images at the domain level but encourages the network to focus on the identity features instead of facial details, such as poses and accessories. Extensive experiments conducted on four challenging NIR-VIS face recognition benchmarks demonstrate that the proposed method can achieve comparable performance with the state-of-the-art (SOTA) methods without requiring any existing NIR-VIS face recognition datasets. With slightly fine-tuning on the target NIR-VIS face recognition datasets, our method can significantly surpass the SOTA performance. Code and pretrained models are released under the insightface (https://github.com/deepinsight/insightface/tree/master/recognition).