# Compact Deep Aggregation for Set Retrieval

arXiv: https://arxiv.org/abs/2003.11794 · HTML: https://mlsanity.com/paper/2003.11794/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Mar 2020.

## Built on by (most cited first, 8 of 8)

- [FaceSigns: Semi-Fragile Neural Watermarks for Media Authentication and Countering Deepfakes](https://mlsanity.com/paper/2204.01960/) (2022)
- [WithAnyone: Towards Controllable and ID Consistent Image Generation](https://mlsanity.com/paper/2510.14975/) (2025)
- [On permutation-invariant neural networks](https://mlsanity.com/paper/2403.17410/) (2024)
- [Self-supervised Video Retrieval Transformer Network](https://mlsanity.com/paper/2104.07993/) (2021)
- [Weakly Supervised Face Naming with Symmetry-Enhanced Contrastive Loss](https://mlsanity.com/paper/2210.08957/) (2022)
- [Advances in Set Function Learning: A Survey of Techniques and Applications](https://mlsanity.com/paper/2501.14991/) (2025)
- [AggNet: Learning to Aggregate Faces for Group Membership Verification](https://mlsanity.com/paper/2206.08683/) (2022)
- [M3-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering](https://mlsanity.com/paper/2604.25122/) (2026)

## Abstract (arXiv)

The objective of this work is to learn a compact embedding of a set of descriptors that is suitable for efficient retrieval and ranking, whilst maintaining discriminability of the individual descriptors. We focus on a specific example of this general problem -- that of retrieving images containing multiple faces from a large scale dataset of images. Here the set consists of the face descriptors in each image, and given a query for multiple identities, the goal is then to retrieve, in order, images which contain all the identities, all but one, etc To this end, we make the following contributions: first, we propose a CNN architecture -- em SetNet -- to achieve the objective: it learns face descriptors and their aggregation over a set to produce a compact fixed length descriptor designed for set retrieval, and the score of an image is a count of the number of identities that match the query; second, we show that this compact descriptor has minimal loss of discriminability up to two faces per image, and degrades slowly after that -- far exceeding a number of baselines; third, we explore the speed vs. retrieval quality trade-off for set retrieval using this compact descriptor; and, finally, we collect and annotate a large dataset of images containing various number of celebrities, which we use for evaluation and is publicly released.