# On the Reliability of Biometric Datasets: How Much Test Data Ensures Reliability?

arXiv: https://arxiv.org/abs/2501.06504 · HTML: https://mlsanity.com/paper/2501.06504/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2025.

## Built on by (most cited first, 5 of 5)

- [On the Illusion of Gender Bias in Face Recognition: Explaining the Fairness Issue Through Non-demographic Attributes](https://mlsanity.com/paper/2501.12020/) (2025)
- [A Comprehensive Re-Evaluation of Biometric Modality Properties in the Modern Era](https://mlsanity.com/paper/2508.13874/) (2025)
- [A Responsible Face Recognition Approach for Small and Mid-Scale Systems Through Personalized Neural Networks](https://mlsanity.com/paper/2505.19920/) (2025)
- [SoK: Security Evaluation of Wi-Fi CSI Biometrics: Attacks, Metrics, and Open Challenges](https://mlsanity.com/paper/2511.11381/) (2025)
- [Learning to Attract and Repel: Dual Quality Margin Learning for Face Recognition (DQM-Face)](https://mlsanity.com/paper/2609.02644/) (2026)

## Abstract (arXiv)

Biometric authentication is increasingly popular for its convenience and accuracy. However, while recent advancements focus on reducing errors and expanding modalities, the reliability of reported performance metrics often remains overlooked. Understanding reliability is critical, as it communicates how accurately reported error rates represent a system's actual performance, considering the uncertainty in error-rate estimates from test data. Currently, there is no widely accepted standard for reporting these uncertainties and indeed biometric studies rarely provide reliability estimates, limiting comparability and interpretation. To address this gap, we introduce BioQuake--a measure to estimate uncertainty in biometric verification systems--and empirically validate it on four systems and three datasets. Based on BioQuake, we provide simple guidelines for estimating performance uncertainty and facilitating reliable reporting. Additionally, we apply BioQuake to analyze biometric recognition performance on 62 biometric datasets used in research across eight modalities: face, fingerprint, gait, iris, keystroke, eye movement, Electroencephalogram (EEG), and Electrocardiogram (ECG). Our analysis shows that reported state-of-the-art performance often deviates significantly from actual error rates, potentially leading to inaccurate conclusions. To support researchers and foster the development of more reliable biometric systems and datasets, we release BioQuake as an easy-to-use web tool for reliability calculations.