# FaceCoresetNet: Differentiable Coresets for Face Set Recognition

arXiv: https://arxiv.org/abs/2308.14075 · HTML: https://mlsanity.com/paper/2308.14075/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 27 Aug 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2103.04098/) (2021)
- [Modeling Point Clouds with Self-Attention and Gumbel Subset Sampling](https://mlsanity.com/paper/1904.03375/) (2019)
- [Efficient DETR: Improving End-to-End Object Detector with Dense Prior](https://mlsanity.com/paper/2104.01318/) (2021)
- [Revisiting Training Strategies and Generalization Performance in Deep Metric Learning](https://mlsanity.com/paper/2002.08473/) (2020)
- [Bayesian Coreset Construction via Greedy Iterative Geodesic Ascent](https://mlsanity.com/paper/1802.01737/) (2018)
- [Adaptive Second Order Coresets for Data-efficient Machine Learning](https://mlsanity.com/paper/2207.13887/) (2022)
- [Small-GAN: Speeding Up GAN Training Using Core-sets](https://mlsanity.com/paper/1910.13540/) (2019)
- [Quality Aware Network for Set to Set Recognition](https://mlsanity.com/paper/1704.03373/) (2017)
- [In Defense of Core-set: A Density-aware Core-set Selection for Active Learning](https://mlsanity.com/paper/2206.04838/) (2022)
- [Cluster and Aggregate: Face Recognition with Large Probe Set](https://mlsanity.com/paper/2210.10864/) (2022)
- [Data-Independent Structured Pruning of Neural Networks via Coresets](https://mlsanity.com/paper/2008.08316/) (2020)
- [Multicolumn Networks for Face Recognition](https://mlsanity.com/paper/1807.09192/) (2018)
- [Video Face Recognition: Component-wise Feature Aggregation Network (C-FAN)](https://mlsanity.com/paper/1902.07327/) (2019)
- [Dependency-aware Attention Control for Unconstrained Face Recognition with Image Sets](https://mlsanity.com/paper/1907.03030/) (2019)
- [Permutation-invariant Feature Restructuring for Correlation-aware Image Set-based Recognition](https://mlsanity.com/paper/1908.01174/) (2019)
- [Unconstrained Still/Video-Based Face Verification with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1605.02686/) (2016)
- [Set-Based Face Recognition Beyond Disentanglement: Burstiness Suppression With Variance Vocabulary](https://mlsanity.com/paper/2304.06249/) (2023)

## Built on by (most cited first, 1 of 1)

- [An Application-Agnostic Automatic Target Recognition System Using Vision Language Models](https://mlsanity.com/paper/2411.03491/) (2024)

## Abstract (arXiv)

In set-based face recognition, we aim to compute the most discriminative descriptor from an unbounded set of images and videos showing a single person. A discriminative descriptor balances two policies when aggregating information from a given set. The first is a quality-based policy: emphasizing high-quality and down-weighting low-quality images. The second is a diversity-based policy: emphasizing unique images in the set and down-weighting multiple occurrences of similar images as found in video clips which can overwhelm the set representation. This work frames face-set representation as a differentiable coreset selection problem. Our model learns how to select a small coreset of the input set that balances quality and diversity policies using a learned metric parameterized by the face quality, optimized end-to-end. The selection process is a differentiable farthest-point sampling (FPS) realized by approximating the non-differentiable Argmax operation with differentiable sampling from the Gumbel-Softmax distribution of distances. The small coreset is later used as queries in a self and cross-attention architecture to enrich the descriptor with information from the whole set. Our model is order-invariant and linear in the input set size. We set a new SOTA to set face verification on the IJB-B and IJB-C datasets. Our code is publicly available.