# Asymmetric Image Retrieval with Cross Model Compatible Ensembles

arXiv: https://arxiv.org/abs/2303.17531 · HTML: https://mlsanity.com/paper/2303.17531/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 61 references that are in the map.
- First posted on arXiv 30 Mar 2023.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Fine-tuning CNN Image Retrieval with No Human Annotation](https://mlsanity.com/paper/1711.02512/) (2017)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Circle Loss: A Unified Perspective of Pair Similarity Optimization](https://mlsanity.com/paper/2002.10857/) (2020)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [NormFace: L2 Hypersphere Embedding for Face Verification](https://mlsanity.com/paper/1704.06369/) (2017)
- [Deep Learning Face Representation by Joint Identification-Verification](https://mlsanity.com/paper/1406.4773/) (2014)
- [Towards Backward-Compatible Representation Learning](https://mlsanity.com/paper/2003.11942/) (2020)
- [Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation](https://mlsanity.com/paper/1810.11750/) (2018)
- [Learning and aggregating deep local descriptors for instance-level recognition](https://mlsanity.com/paper/2007.13172/) (2020)
- [The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1704.01664/) (2017)
- [Visual Instance Retrieval with Deep Convolutional Networks](https://mlsanity.com/paper/1412.6574/) (2014)
- [Local Features and Visual Words Emerge in Activations](https://mlsanity.com/paper/1905.06358/) (2019)
- [Learning Compatible Embeddings](https://mlsanity.com/paper/2108.01958/) (2021)
- [Unified Representation Learning for Cross Model Compatibility](https://mlsanity.com/paper/2008.04821/) (2020)
- [Asymmetric metric learning for knowledge transfer](https://mlsanity.com/paper/2006.16331/) (2020)
- [Compatibility-aware Heterogeneous Visual Search](https://mlsanity.com/paper/2105.06047/) (2021)
- [Towards Universal Backward-Compatible Representation Learning](https://mlsanity.com/paper/2203.01583/) (2022)
- [BT2: Backward-compatible Training with Basis Transformation](https://mlsanity.com/paper/2211.03989/) (2022)
- [FastFill: Efficient Compatible Model Update](https://mlsanity.com/paper/2303.04766/) (2023)

## Built on by (most cited first, 2 of 2)

- [Data Pruning via Separability, Integrity, and Model Uncertainty-Aware Importance Sampling](https://mlsanity.com/paper/2409.13915/) (2024)
- [Learning Compatible Multi-Prize Subnetworks for Asymmetric Retrieval](https://mlsanity.com/paper/2504.11879/) (2025)

## Abstract (arXiv)

The asymmetrical retrieval setting is a well suited solution for resource constrained applications such as face recognition and image retrieval. In this setting, a large model is used for indexing the gallery while a lightweight model is used for querying. The key principle in such systems is ensuring that both models share the same embedding space. Most methods in this domain are based on knowledge distillation. While useful, they suffer from several drawbacks: they are upper-bounded by the performance of the single best model found and cannot be extended to use an ensemble of models in a straightforward manner. In this paper we present an approach that does not rely on knowledge distillation, rather it utilizes embedding transformation models. This allows the use of N independently trained and diverse gallery models (e.g., trained on different datasets or having a different architecture) and a single query model. As a result, we improve the overall accuracy beyond that of any single model while maintaining a low computational budget for querying. Additionally, we propose a gallery image rejection method that utilizes the diversity between multiple transformed embeddings to estimate the uncertainty of gallery images.