# Part-based Face Recognition with Vision Transformers

arXiv: https://arxiv.org/abs/2212.00057 · HTML: https://mlsanity.com/paper/2212.00057/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 76 references that are in the map.
- First posted on arXiv 30 Nov 2022.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014)
- [Circle Loss: A Unified Perspective of Pair Similarity Optimization](https://mlsanity.com/paper/2002.10857/) (2020)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition](https://mlsanity.com/paper/2004.00288/) (2020)
- [Large-Margin Softmax Loss for Convolutional Neural Networks](https://mlsanity.com/paper/1612.02295/) (2016)
- [Incorporating Convolution Designs into Visual Transformers](https://mlsanity.com/paper/2103.11816/) (2021)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)

## Built on by (most cited first, 18 of 18)

- [FRoundation: Are Foundation Models Ready for Face Recognition?](https://mlsanity.com/paper/2410.23831/) (2024)
- [50 Years of Automated Face Recognition](https://mlsanity.com/paper/2505.24247/) (2025)
- [LAFS: Landmark-based Facial Self-supervised Learning for Face Recognition](https://mlsanity.com/paper/2403.08161/) (2024)
- [LVFace: Progressive Cluster Optimization for Large Vision Models in Face Recognition](https://mlsanity.com/paper/2501.13420/) (2025)
- [CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition](https://mlsanity.com/paper/2409.18876/) (2024)
- [ViT-FIQA: Assessing Face Image Quality using Vision Transformers](https://mlsanity.com/paper/2508.13957/) (2025)
- [ViTNT-FIQA: Training-Free Face Image Quality Assessment with Vision Transformers](https://mlsanity.com/paper/2601.05741/) (2026)
- [RepFace: Refining Closed-Set Noise with Progressive Label Correction for Face Recognition](https://mlsanity.com/paper/2412.12031/) (2024)
- [FaceGCD: Generalized Face Discovery via Dynamic Prefix Generation](https://mlsanity.com/paper/2507.22353/) (2025)
- [Nested-TNT: Hierarchical Vision Transformers with Multi-Scale Feature Processing](https://mlsanity.com/paper/2404.13434/) (2024)
- [Quo Vadis RankList-based System in Face Recognition?](https://mlsanity.com/paper/2410.01498/) (2024)
- [Unified Face Matching and Physical-Digital Spoofing Attack Detection](https://mlsanity.com/paper/2501.09635/) (2025)
- [A Responsible Face Recognition Approach for Small and Mid-Scale Systems Through Personalized Neural Networks](https://mlsanity.com/paper/2505.19920/) (2025)
- [A survey of facial recognition techniques](https://mlsanity.com/paper/2601.06239/) (2026)
- [AHAN: Asymmetric Hierarchical Attention Network for Identical Twin Face Verification](https://mlsanity.com/paper/2602.21503/) (2026)
- [ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptation](https://mlsanity.com/paper/2606.12023/) (2026)
- [Vision Transformers for Face Recognition Need More Registers](https://mlsanity.com/paper/2606.12036/) (2026)
- [Bias Mitigation in Face Recognition via Demographic-based Supervised Contrastive Learning](https://mlsanity.com/paper/2608.12971/) (2026)

## Abstract (arXiv)

Holistic methods using CNNs and margin-based losses have dominated research on face recognition. In this work, we depart from this setting in two ways: (a) we employ the Vision Transformer as an architecture for training a very strong baseline for face recognition, simply called fViT, which already surpasses most state-of-the-art face recognition methods. (b) Secondly, we capitalize on the Transformer's inherent property to process information (visual tokens) extracted from irregular grids to devise a pipeline for face recognition which is reminiscent of part-based face recognition methods. Our pipeline, called part fViT, simply comprises a lightweight network to predict the coordinates of facial landmarks followed by the Vision Transformer operating on patches extracted from the predicted landmarks, and it is trained end-to-end with no landmark supervision. By learning to extract discriminative patches, our part-based Transformer further boosts the accuracy of our Vision Transformer baseline achieving state-of-the-art accuracy on several face recognition benchmarks.