# Balancing Tails when Comparing Distributions: Comprehensive Equity Index (CEI) with Application to Bias Evaluation in Operational Face Biometrics

arXiv: https://arxiv.org/abs/2506.10564 · HTML: https://mlsanity.com/paper/2506.10564/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 54 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2103.04098/) (2021)
- [Partial FC: Training 10 Million Identities on a Single Machine](https://mlsanity.com/paper/2010.05222/) (2020)
- [Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learning](https://mlsanity.com/paper/1911.10692/) (2019)
- [Artificial Intelligence: the global landscape of ethics guidelines](https://mlsanity.com/paper/1906.11668/) (2019)
- [A Comprehensive Study on Face Recognition Biases Beyond Demographics](https://mlsanity.com/paper/2103.01592/) (2021)
- [SensitiveLoss: Improving Accuracy and Fairness of Face Representations with Discrimination-Aware Deep Learning](https://mlsanity.com/paper/2004.11246/) (2020)
- [GANDiffFace: Controllable Generation of Synthetic Datasets for Face Recognition with Realistic Variations](https://mlsanity.com/paper/2305.19962/) (2023)
- [Fairness in Biometrics: a figure of merit to assess biometric verification systems](https://mlsanity.com/paper/2011.02395/) (2020)
- [Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitment](https://mlsanity.com/paper/2302.10908/) (2023)
- [InsideBias: Measuring Bias in Deep Networks and Application to Face Gender Biometrics](https://mlsanity.com/paper/2004.06592/) (2020)
- [FRCSyn Challenge at WACV 2024:Face Recognition Challenge in the Era of Synthetic Data](https://mlsanity.com/paper/2311.10476/) (2023)
- [Algorithmic Discrimination: Formulation and Exploration in Deep Learning-based Face Biometrics](https://mlsanity.com/paper/1912.01842/) (2019)
- [Facial Soft Biometrics for Recognition in the Wild: Recent Works, Annotation, and COTS Evaluation](https://mlsanity.com/paper/2210.13129/) (2022)
- [Evaluating Proposed Fairness Models for Face Recognition Algorithms](https://mlsanity.com/paper/2203.05051/) (2022)
- [Is My Data in Your AI? Membership Inference Test (MINT) applied to Face Biometrics](https://mlsanity.com/paper/2402.09225/) (2024)
- [SetMargin Loss applied to Deep Keystroke Biometrics with Circle Packing Interpretation](https://mlsanity.com/paper/2109.00938/) (2021)
- [IFBiD: Inference-Free Bias Detection](https://mlsanity.com/paper/2109.04374/) (2021)
- [Is My Text in Your AI Model? Gradient-based Membership Inference Test applied to LLMs](https://mlsanity.com/paper/2503.07384/) (2025)
- [Cross-Sensor Periocular Biometrics in a Global Pandemic: Comparative Benchmark and Novel Multialgorithmic Approach](https://mlsanity.com/paper/1902.08123/) (2019)
- [FaceQvec: Vector Quality Assessment for Face Biometrics based on ISO Compliance](https://mlsanity.com/paper/2111.02078/) (2021)
- [Review of Demographic Fairness in Face Recognition](https://mlsanity.com/paper/2502.02309/) (2025)
- [Keystroke Verification Challenge (KVC): Biometric and Fairness Benchmark Evaluation](https://mlsanity.com/paper/2311.06000/) (2023)
- [Learning Emotional-Blinded Face Representations](https://mlsanity.com/paper/2009.08704/) (2020)
- [Fairness Under Cover: Evaluating the Impact of Occlusions on Demographic Bias in Facial Recognition](https://mlsanity.com/paper/2408.10175/) (2024)
- [Comprehensive Equity Index (CEI): Definition and Application to Bias Evaluation in Biometrics](https://mlsanity.com/paper/2409.01928/) (2024)

## Built on by (most cited first, 2 of 2)

- [Auditing Training Data in Domain-adapted LLMs: LoRA-MINT](https://mlsanity.com/paper/2606.06946/) (2026)
- [Unraveling Machine Behavior by Multi-Level Bias Analysis and Detection: Methodology and Application to Computer Vision](https://mlsanity.com/paper/2607.07236/) (2026)

## Abstract (arXiv)

Demographic bias in high-performance face recognition (FR) systems often eludes detection by existing metrics, especially with respect to subtle disparities in the tails of the score distribution. We introduce the Comprehensive Equity Index (CEI), a novel metric designed to address this limitation. CEI uniquely analyzes genuine and impostor score distributions separately, enabling a configurable focus on tail probabilities while also considering overall distribution shapes. Our extensive experiments (evaluating state-of-the-art FR systems, intentionally biased models, and diverse datasets) confirm CEI's superior ability to detect nuanced biases where previous methods fall short. Furthermore, we present CEI^A, an automated version of the metric that enhances objectivity and simplifies practical application. CEI provides a robust and sensitive tool for operational FR fairness assessment. The proposed methods have been developed particularly for bias evaluation in face biometrics but, in general, they are applicable for comparing statistical distributions in any problem where one is interested in analyzing the distribution tails.