# Double Trouble? Impact and Detection of Duplicates in Face Image Datasets

arXiv: https://arxiv.org/abs/2401.14088 · HTML: https://mlsanity.com/paper/2401.14088/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 16 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Builds on (most cited first, 7 of 7)

- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014)
- [Sample and Computation Redistribution for Efficient Face Detection](https://mlsanity.com/paper/2105.04714/) (2021)
- [CR-FIQA: Face Image Quality Assessment by Learning Sample Relative Classifiability](https://mlsanity.com/paper/2112.06592/) (2021)
- [Low-Resolution Face Recognition](https://mlsanity.com/paper/1811.08965/) (2018)
- [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.07698/) (2018)
- [Considerations on the Evaluation of Biometric Quality Assessment Algorithms](https://mlsanity.com/paper/2303.13294/) (2023)

## Built on by (most cited first, 2 of 2)

- [Is My Data in Your AI? Membership Inference Test (MINT) applied to Face Biometrics](https://mlsanity.com/paper/2402.09225/) (2024)
- [Identity Overlap Between Face Recognition Train/Test Data: Causing Optimistic Bias in Accuracy Measurement](https://mlsanity.com/paper/2405.09403/) (2024)

## Abstract (arXiv)

Various face image datasets intended for facial biometrics research were created via web-scraping, i.e. the collection of images publicly available on the internet. This work presents an approach to detect both exactly and nearly identical face image duplicates, using file and image hashes. The approach is extended through the use of face image preprocessing. Additional steps based on face recognition and face image quality assessment models reduce false positives, and facilitate the deduplication of the face images both for intra- and inter-subject duplicate sets. The presented approach is applied to five datasets, namely LFW, TinyFace, Adience, CASIA-WebFace, and C-MS-Celeb (a cleaned MS-Celeb-1M variant). Duplicates are detected within every dataset, with hundreds to hundreds of thousands of duplicates for all except LFW. Face recognition and quality assessment experiments indicate a minor impact on the results through the duplicate removal. The final deduplication data is publicly available.