# Towards NIR-VIS Masked Face Recognition

arXiv: https://arxiv.org/abs/2104.06761 · HTML: https://mlsanity.com/paper/2104.06761/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices](https://mlsanity.com/paper/1804.07573/) (2018)
- [FaceBoxes: A CPU Real-time Face Detector with High Accuracy](https://mlsanity.com/paper/1708.05234/) (2017)
- [Wasserstein CNN: Learning Invariant Features for NIR-VIS Face Recognition](https://mlsanity.com/paper/1708.02412/) (2017)
- [Not Afraid of the Dark: NIR-VIS Face Recognition via Cross-spectral Hallucination and Low-rank Embedding](https://mlsanity.com/paper/1611.06638/) (2016)
- [Polarimetric Thermal to Visible Face Verification via Self-Attention Guided Synthesis](https://mlsanity.com/paper/1904.07344/) (2019)
- [Semi-Siamese Training for Shallow Face Learning](https://mlsanity.com/paper/2007.08398/) (2020)
- [Cross-Spectral Face Hallucination via Disentangling Independent Factors](https://mlsanity.com/paper/1909.04365/) (2019)
- [Adversarial Discriminative Heterogeneous Face Recognition](https://mlsanity.com/paper/1709.03675/) (2017)
- [Multi-Scale Thermal to Visible Face Verification via Attribute Guided Synthesis](https://mlsanity.com/paper/2004.09502/) (2020)
- [Polarimetric Thermal to Visible Face Verification via Attribute Preserved Synthesis](https://mlsanity.com/paper/1901.00889/) (2019)

## Built on by (most cited first, 5 of 5)

- [WebFace260M: A Benchmark for Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2204.10149/) (2022)
- [Masked Face Recognition Challenge: The WebFace260M Track Report](https://mlsanity.com/paper/2108.07189/) (2021)
- [MLFW: A Database for Face Recognition on Masked Faces](https://mlsanity.com/paper/2109.05804/) (2021)
- [A Survey on Masked Facial Detection Methods and Datasets for Fighting Against COVID-19](https://mlsanity.com/paper/2201.04777/) (2022)
- [Multi-Dataset Benchmarks for Masked Identification using Contrastive Representation Learning](https://mlsanity.com/paper/2106.05596/) (2021)

## Abstract (arXiv)

Near-infrared to visible (NIR-VIS) face recognition is the most common case in heterogeneous face recognition, which aims to match a pair of face images captured from two different modalities. Existing deep learning based methods have made remarkable progress in NIR-VIS face recognition, while it encounters certain newly-emerged difficulties during the pandemic of COVID-19, since people are supposed to wear facial masks to cut off the spread of the virus. We define this task as NIR-VIS masked face recognition, and find it problematic with the masked face in the NIR probe image. First, the lack of masked face data is a challenging issue for the network training. Second, most of the facial parts (cheeks, mouth, nose etc.) are fully occluded by the mask, which leads to a large amount of loss of information. Third, the domain gap still exists in the remaining facial parts. In such scenario, the existing methods suffer from significant performance degradation caused by the above issues. In this paper, we aim to address the challenge of NIR-VIS masked face recognition from the perspectives of training data and training method. Specifically, we propose a novel heterogeneous training method to maximize the mutual information shared by the face representation of two domains with the help of semi-siamese networks. In addition, a 3D face reconstruction based approach is employed to synthesize masked face from the existing NIR image. Resorting to these practices, our solution provides the domain-invariant face representation which is also robust to the mask occlusion. Extensive experiments on three NIR-VIS face datasets demonstrate the effectiveness and cross-dataset-generalization capacity of our method.