# Fast and Interpretable Face Identification for Out-Of-Distribution Data Using Vision Transformers

arXiv: https://arxiv.org/abs/2311.02803 · HTML: https://mlsanity.com/paper/2311.02803/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 62 references that are in the map.
- First posted on arXiv 6 Nov 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](https://mlsanity.com/paper/2103.15679/) (2021)
- [Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments](https://mlsanity.com/paper/1708.08197/) (2017)
- [Better plain ViT baselines for ImageNet-1k](https://mlsanity.com/paper/2205.01580/) (2022)
- [Towards Transferable Adversarial Attack against Deep Face Recognition](https://mlsanity.com/paper/2004.05790/) (2020)
- [NFormer: Robust Person Re-identification with Neighbor Transformer](https://mlsanity.com/paper/2204.09331/) (2022)
- [Face Transformer for Recognition](https://mlsanity.com/paper/2103.14803/) (2021)
- [TransMatcher: Deep Image Matching Through Transformers for Generalizable Person Re-identification](https://mlsanity.com/paper/2105.14432/) (2021)
- [Towards Interpretable Deep Metric Learning with Structural Matching](https://mlsanity.com/paper/2108.05889/) (2021)
- [OODformer: Out-Of-Distribution Detection Transformer](https://mlsanity.com/paper/2107.08976/) (2021)
- [Visualizing Deep Similarity Networks](https://mlsanity.com/paper/1901.00536/) (2019)
- [End2End Occluded Face Recognition by Masking Corrupted Features](https://mlsanity.com/paper/2108.09468/) (2021)

## Built on by (most cited first, 4 of 4)

- [Exploring the Synergies of Hybrid CNNs and ViTs Architectures for Computer Vision: A survey](https://mlsanity.com/paper/2402.02941/) (2024)
- [FaceTracer: Unveiling Source Identities from Swapped Face Images and Videos for Fraud Prevention](https://mlsanity.com/paper/2412.08082/) (2024)
- [TAB: Transformer Attention Bottlenecks enable User Intervention and Debugging in Vision-Language Models](https://mlsanity.com/paper/2412.18675/) (2024)
- [MGFace: Mask-Gated Face Matching via Conditional Similarity Routing](https://mlsanity.com/paper/2607.13187/) (2026)

## Abstract (arXiv)

Most face identification approaches employ a Siamese neural network to compare two images at the image embedding level. Yet, this technique can be subject to occlusion (e.g. faces with masks or sunglasses) and out-of-distribution data. DeepFace-EMD (Phan et al. 2022) reaches state-of-the-art accuracy on out-of-distribution data by first comparing two images at the image level, and then at the patch level. Yet, its later patch-wise re-ranking stage admits a large O(n3 log n) time complexity (for n patches in an image) due to the optimal transport optimization. In this paper, we propose a novel, 2-image Vision Transformers (ViTs) that compares two images at the patch level using cross-attention. After training on 2M pairs of images on CASIA Webface (Yi et al. 2014), our model performs at a comparable accuracy as DeepFace-EMD on out-of-distribution data, yet at an inference speed more than twice as fast as DeepFace-EMD (Phan et al. 2022). In addition, via a human study, our model shows promising explainability through the visualization of cross-attention. We believe our work can inspire more explorations in using ViTs for face identification.