# FACESEC: A Fine-grained Robustness Evaluation Framework for Face Recognition Systems

arXiv: https://arxiv.org/abs/2104.04107 · HTML: https://mlsanity.com/paper/2104.04107/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Delving into Transferable Adversarial Examples and Black-box Attacks](https://mlsanity.com/paper/1611.02770/) (2016)
- [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014)
- [Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples](https://mlsanity.com/paper/1605.07277/) (2016)
- [Large-Margin Softmax Loss for Convolutional Neural Networks](https://mlsanity.com/paper/1612.02295/) (2016)
- [Efficient Decision-based Black-box Adversarial Attacks on Face Recognition](https://mlsanity.com/paper/1904.04433/) (2019)
- [Deep Learning Face Representation by Joint Identification-Verification](https://mlsanity.com/paper/1406.4773/) (2014)
- [SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing](https://mlsanity.com/paper/1906.07927/) (2019)
- [Defending Against Physically Realizable Attacks on Image Classification](https://mlsanity.com/paper/1909.09552/) (2019)

## Built on by (most cited first, 14 of 14)

- [Towards Effective Adversarial Textured 3D Meshes on Physical Face Recognition](https://mlsanity.com/paper/2303.15818/) (2023)
- [Fair SA: Sensitivity Analysis for Fairness in Face Recognition](https://mlsanity.com/paper/2202.03586/) (2022)
- [Certifiably Robust Interpretation via Renyi Differential Privacy](https://mlsanity.com/paper/2107.01561/) (2021)
- [Controllable Evaluation and Generation of Physical Adversarial Patch on Face Recognition](https://mlsanity.com/paper/2203.04623/) (2022)
- [Detecting Adversarial Faces Using Only Real Face Self-Perturbations](https://mlsanity.com/paper/2304.11359/) (2023)
- [Exposing Fine-Grained Adversarial Vulnerability of Face Anti-Spoofing Models](https://mlsanity.com/paper/2205.14851/) (2022)
- [AdvFAS: A robust face anti-spoofing framework against adversarial examples](https://mlsanity.com/paper/2308.02116/) (2023)
- [A Survey of Face Recognition](https://mlsanity.com/paper/2212.13038/) (2022)
- [Semantic Adversarial Attacks on Face Recognition through Significant Attributes](https://mlsanity.com/paper/2301.12046/) (2023)
- [Detecting Adversarial Data using Perturbation Forgery](https://mlsanity.com/paper/2405.16226/) (2024)
- [Watch Out for the Confusing Faces: Detecting Face Swapping with the Probability Distribution of Face Identification Models](https://mlsanity.com/paper/2303.13131/) (2023)
- [FaceCat: Enhancing Face Recognition Security with a Unified Diffusion Model](https://mlsanity.com/paper/2404.09193/) (2024)
- [Sy-FAR: Symmetry-based Fair Adversarial Robustness](https://mlsanity.com/paper/2509.12939/) (2025)
- [RobustGait: Robustness Analysis for Appearance Based Gait Recognition](https://mlsanity.com/paper/2511.13065/) (2025)

## Abstract (arXiv)

We present FACESEC, a framework for fine-grained robustness evaluation of face recognition systems. FACESEC evaluation is performed along four dimensions of adversarial modeling: the nature of perturbation (e.g., pixel-level or face accessories), the attacker's system knowledge (about training data and learning architecture), goals (dodging or impersonation), and capability (tailored to individual inputs or across sets of these). We use FACESEC to study five face recognition systems in both closed-set and open-set settings, and to evaluate the state-of-the-art approach for defending against physically realizable attacks on these. We find that accurate knowledge of neural architecture is significantly more important than knowledge of the training data in black-box attacks. Moreover, we observe that open-set face recognition systems are more vulnerable than closed-set systems under different types of attacks. The efficacy of attacks for other threat model variations, however, appears highly dependent on both the nature of perturbation and the neural network architecture. For example, attacks that involve adversarial face masks are usually more potent, even against adversarially trained models, and the ArcFace architecture tends to be more robust than the others.