# KeyPoint Relative Position Encoding for Face Recognition

arXiv: https://arxiv.org/abs/2403.14852 · HTML: https://mlsanity.com/paper/2403.14852/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 91 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Builds on (most cited first, 30 of 68)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Deep Learning for Person Re-identification: A Survey and Outlook](https://mlsanity.com/paper/2001.04193/) (2020)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition](https://mlsanity.com/paper/2004.00288/) (2020)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments](https://mlsanity.com/paper/1708.08197/) (2017)
- [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2103.04098/) (2021)

## Built on by (most cited first, 30 of 38)

- [FRoundation: Are Foundation Models Ready for Face Recognition?](https://mlsanity.com/paper/2410.23831/) (2024)
- [BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Models](https://mlsanity.com/paper/2505.18132/) (2025)
- [50 Years of Automated Face Recognition](https://mlsanity.com/paper/2505.24247/) (2025)
- [SapiensID: Foundation for Human Recognition](https://mlsanity.com/paper/2504.04708/) (2025)
- [Person Recognition at Altitude and Range: Fusion of Face, Body Shape and Gait](https://mlsanity.com/paper/2505.04616/) (2025)
- [Open-Set Biometrics: Beyond Good Closed-Set Models](https://mlsanity.com/paper/2407.16133/) (2024)
- [Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning](https://mlsanity.com/paper/2504.07198/) (2025)
- [A Quality-Guided Mixture of Score-Fusion Experts Framework for Human Recognition](https://mlsanity.com/paper/2508.00053/) (2025)
- [HAMoBE: Hierarchical and Adaptive Mixture of Biometric Experts for Video-based Person ReID](https://mlsanity.com/paper/2508.05038/) (2025)
- [ViT-FIQA: Assessing Face Image Quality using Vision Transformers](https://mlsanity.com/paper/2508.13957/) (2025)
- [BlurDM: A Blur Diffusion Model for Image Deblurring](https://mlsanity.com/paper/2512.03979/) (2025)
- [ViTNT-FIQA: Training-Free Face Image Quality Assessment with Vision Transformers](https://mlsanity.com/paper/2601.05741/) (2026)
- [LocalScore: Local Density-Aware Similarity Scoring for Biometrics](https://mlsanity.com/paper/2602.01012/) (2026)
- [FusionAgent: A Multimodal Agent with Dynamic Model Selection for Human Recognition](https://mlsanity.com/paper/2603.26908/) (2026)
- [IDFace: Face Template Protection for Efficient and Secure Identification](https://mlsanity.com/paper/2507.12050/) (2025)
- [Non-Adaptive Adversarial Face Generation](https://mlsanity.com/paper/2507.12107/) (2025)
- [MLLM-based Textual Explanations for Face Comparison](https://mlsanity.com/paper/2603.16629/) (2026)
- [LH2Face: Loss function for Hard High-quality Face](https://mlsanity.com/paper/2506.23555/) (2025)
- [IDSelect: A RL-Based Cost-Aware Selection Agent for Video-based Multi-Modal Person Recognition](https://mlsanity.com/paper/2602.18990/) (2026)
- [Compatibility of Face Embeddings Across Deep Neural Networks](https://mlsanity.com/paper/2604.07282/) (2026)
- [HNDiff: Haze-Noise Diffusion for Image Dehazing](https://mlsanity.com/paper/2608.10995/) (2026)
- [HeadsUp! High-Fidelity Portrait Image Super-Resolution](https://mlsanity.com/paper/2510.09924/) (2025)
- [Restricted Receptive Fields for Face Verification](https://mlsanity.com/paper/2510.10753/) (2025)
- [ATTN-FIQA: Interpretable Attention-based Face Image Quality Assessment with Vision Transformers](https://mlsanity.com/paper/2604.22841/) (2026)
- [EX-FIQA: Leveraging Intermediate Early eXit Representations from Vision Transformers for Face Image Quality Assessment](https://mlsanity.com/paper/2604.22842/) (2026)
- [ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptation](https://mlsanity.com/paper/2606.12023/) (2026)
- [Vision Transformers for Face Recognition Need More Registers](https://mlsanity.com/paper/2606.12036/) (2026)
- [Optimizing Image Preparation and Compression for Face Recognition within 1024 Bytes](https://mlsanity.com/paper/2606.30321/) (2026)
- [Subtoken Vision Transformer for Fine-grained Recognition](https://mlsanity.com/paper/2607.09086/) (2026)
- [MagicMakeup: A Region-Controllable Diffusion Transformer for High-Fidelity Makeup-Transfer](https://mlsanity.com/paper/2607.20924/) (2026)

## Abstract (arXiv)

In this paper, we address the challenge of making ViT models more robust to unseen affine transformations. Such robustness becomes useful in various recognition tasks such as face recognition when image alignment failures occur. We propose a novel method called KP-RPE, which leverages key points (e.g.~facial landmarks) to make ViT more resilient to scale, translation, and pose variations. We begin with the observation that Relative Position Encoding (RPE) is a good way to bring affine transform generalization to ViTs. RPE, however, can only inject the model with prior knowledge that nearby pixels are more important than far pixels. Keypoint RPE (KP-RPE) is an extension of this principle, where the significance of pixels is not solely dictated by their proximity but also by their relative positions to specific keypoints within the image. By anchoring the significance of pixels around keypoints, the model can more effectively retain spatial relationships, even when those relationships are disrupted by affine transformations. We show the merit of KP-RPE in face and gait recognition. The experimental results demonstrate the effectiveness in improving face recognition performance from low-quality images, particularly where alignment is prone to failure. Code and pre-trained models are available.