# Comprehensive Equity Index (CEI): Definition and Application to Bias Evaluation in Biometrics

arXiv: https://arxiv.org/abs/2409.01928 · HTML: https://mlsanity.com/paper/2409.01928/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Deep Face Recognition: A Survey](https://mlsanity.com/paper/1804.06655/) (2018)
- [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2103.04098/) (2021)
- [Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learning](https://mlsanity.com/paper/1911.10692/) (2019)
- [Demographic Bias in Biometrics: A Survey on an Emerging Challenge](https://mlsanity.com/paper/2003.02488/) (2020)
- [A Comprehensive Study on Face Recognition Biases Beyond Demographics](https://mlsanity.com/paper/2103.01592/) (2021)
- [Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulation](https://mlsanity.com/paper/2305.02231/) (2023)
- [SensitiveLoss: Improving Accuracy and Fairness of Face Representations with Discrimination-Aware Deep Learning](https://mlsanity.com/paper/2004.11246/) (2020)
- [Fairness in Biometrics: a figure of merit to assess biometric verification systems](https://mlsanity.com/paper/2011.02395/) (2020)
- [Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitment](https://mlsanity.com/paper/2302.10908/) (2023)
- [InsideBias: Measuring Bias in Deep Networks and Application to Face Gender Biometrics](https://mlsanity.com/paper/2004.06592/) (2020)
- [SDFR: Synthetic Data for Face Recognition Competition](https://mlsanity.com/paper/2404.04580/) (2024)
- [FRCSyn Challenge at WACV 2024:Face Recognition Challenge in the Era of Synthetic Data](https://mlsanity.com/paper/2311.10476/) (2023)
- [SensitiveNets: Learning Agnostic Representations with Application to Face Images](https://mlsanity.com/paper/1902.00334/) (2019)
- [Evaluating Proposed Fairness Models for Face Recognition Algorithms](https://mlsanity.com/paper/2203.05051/) (2022)
- [Facial Expressions as a Vulnerability in Face Recognition](https://mlsanity.com/paper/2011.08809/) (2020)
- [IFBiD: Inference-Free Bias Detection](https://mlsanity.com/paper/2109.04374/) (2021)
- [Introduction to Presentation Attack Detection in Face Biometrics and Recent Advances](https://mlsanity.com/paper/2111.11794/) (2021)
- [Measuring Bias in AI Models: An Statistical Approach Introducing N-Sigma](https://mlsanity.com/paper/2304.13680/) (2023)
- [Periocular Biometrics: A Modality for Unconstrained Scenarios](https://mlsanity.com/paper/2212.13792/) (2022)
- [LivDet2023 -- Fingerprint Liveness Detection Competition: Advancing Generalization](https://mlsanity.com/paper/2309.15578/) (2023)
- [SaFL: Sybil-aware Federated Learning with Application to Face Recognition](https://mlsanity.com/paper/2311.04346/) (2023)

## Built on by (most cited first, 2 of 2)

- [Review of Demographic Fairness in Face Recognition](https://mlsanity.com/paper/2502.02309/) (2025)
- [Balancing Tails when Comparing Distributions: Comprehensive Equity Index (CEI) with Application to Bias Evaluation in Operational Face Biometrics](https://mlsanity.com/paper/2506.10564/) (2025)

## Abstract (arXiv)

We present a novel metric designed, among other applications, to quantify biased behaviors of machine learning models. As its core, the metric consists of a new similarity metric between score distributions that balances both their general shapes and tails' probabilities. In that sense, our proposed metric may be useful in many application areas. Here we focus on and apply it to the operational evaluation of face recognition systems, with special attention to quantifying demographic biases; an application where our metric is especially useful. The topic of demographic bias and fairness in biometric recognition systems has gained major attention in recent years. The usage of these systems has spread in society, raising concerns about the extent to which these systems treat different population groups. A relevant step to prevent and mitigate demographic biases is first to detect and quantify them. Traditionally, two approaches have been studied to quantify differences between population groups in machine learning literature: 1) measuring differences in error rates, and 2) measuring differences in recognition score distributions. Our proposed Comprehensive Equity Index (CEI) trade-offs both approaches combining both errors from distribution tails and general distribution shapes. This new metric is well suited to real-world scenarios, as measured on NIST FRVT evaluations, involving high-performance systems and realistic face databases including a wide range of covariates and demographic groups. We first show the limitations of existing metrics to correctly assess the presence of biases in realistic setups and then propose our new metric to tackle these limitations. We tested the proposed metric with two state-of-the-art models and four widely used databases, showing its capacity to overcome the main flaws of previous bias metrics.