# Compatibility-aware Heterogeneous Visual Search

arXiv: https://arxiv.org/abs/2105.06047 · HTML: https://mlsanity.com/paper/2105.06047/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 13 May 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Fine-tuning CNN Image Retrieval with No Human Annotation](https://mlsanity.com/paper/1711.02512/) (2017)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [NormFace: L2 Hypersphere Embedding for Face Verification](https://mlsanity.com/paper/1704.06369/) (2017)
- [Particular object retrieval with integral max-pooling of CNN activations](https://mlsanity.com/paper/1511.05879/) (2015)
- [DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Images](https://mlsanity.com/paper/1901.07973/) (2019)
- [Blockwisely Supervised Neural Architecture Search with Knowledge Distillation](https://mlsanity.com/paper/1911.13053/) (2019)
- [Towards Backward-Compatible Representation Learning](https://mlsanity.com/paper/2003.11942/) (2020)
- [The Devil of Face Recognition is in the Noise](https://mlsanity.com/paper/1807.11649/) (2018)
- [Visual Instance Retrieval with Deep Convolutional Networks](https://mlsanity.com/paper/1412.6574/) (2014)
- [Cross-dimensional Weighting for Aggregated Deep Convolutional Features](https://mlsanity.com/paper/1512.04065/) (2015)
- [Medical Image Retrieval using Deep Convolutional Neural Network](https://mlsanity.com/paper/1703.08472/) (2017)
- [Unified Representation Learning for Cross Model Compatibility](https://mlsanity.com/paper/2008.04821/) (2020)
- [Asymmetric metric learning for knowledge transfer](https://mlsanity.com/paper/2006.16331/) (2020)
- [Search to Distill: Pearls are Everywhere but not the Eyes](https://mlsanity.com/paper/1911.09074/) (2019)
- [Towards Visual Feature Translation](https://mlsanity.com/paper/1812.00573/) (2018)

## Built on by (most cited first, 23 of 23)

- [CoReS: Compatible Representations via Stationarity](https://mlsanity.com/paper/2111.07632/) (2021)
- [Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2403.01431/) (2024)
- [Boundary-aware Backward-Compatible Representation via Adversarial Learning in Image Retrieval](https://mlsanity.com/paper/2305.02610/) (2023)
- [Dual-Tuning: Joint Prototype Transfer and Structure Regularization for Compatible Feature Learning](https://mlsanity.com/paper/2108.02959/) (2021)
- [Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrieval](https://mlsanity.com/paper/2201.09724/) (2022)
- [FastFill: Efficient Compatible Model Update](https://mlsanity.com/paper/2303.04766/) (2023)
- [Let All be Whitened: Multi-teacher Distillation for Efficient Visual Retrieval](https://mlsanity.com/paper/2312.09716/) (2023)
- [Metric Compatible Training for Online Backfilling in Large-Scale Retrieval](https://mlsanity.com/paper/2301.03767/) (2023)
- [Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion](https://mlsanity.com/paper/2302.02080/) (2023)
- [Binary Embedding-based Retrieval at Tencent](https://mlsanity.com/paper/2302.08714/) (2023)
- [Darwinian Model Upgrades: Model Evolving with Selective Compatibility](https://mlsanity.com/paper/2210.06954/) (2022)
- [Large-to-small Image Resolution Asymmetry in Deep Metric Learning](https://mlsanity.com/paper/2210.05463/) (2022)
- [Backward-Compatible Aligned Representations via an Orthogonal Transformation Layer](https://mlsanity.com/paper/2408.08793/) (2024)
- [IMB-NAS: Neural Architecture Search for Imbalanced Datasets](https://mlsanity.com/paper/2210.00136/) (2022)
- [Asymmetric Image Retrieval with Cross Model Compatible Ensembles](https://mlsanity.com/paper/2303.17531/) (2023)
- [Structure Similarity Preservation Learning for Asymmetric Image Retrieval](https://mlsanity.com/paper/2403.00648/) (2024)
- [Towards Test-time Efficient Visual Place Recognition via Asymmetric Query Processing](https://mlsanity.com/paper/2512.13055/) (2025)
- [An Efficient NAS-based Approach for Handling Imbalanced Datasets](https://mlsanity.com/paper/2406.16972/) (2024)
- [A Stationary (and Therefore Compatible) Representation is All You Need](https://mlsanity.com/paper/2606.12488/) (2026)
- [Towards Regression-Free Neural Networks for Diverse Compute Platforms](https://mlsanity.com/paper/2209.13740/) (2022)
- [Learning Unified Representations for Multi-Resolution Face Recognition](https://mlsanity.com/paper/2310.09563/) (2023)
- [Learning Compatible Multi-Prize Subnetworks for Asymmetric Retrieval](https://mlsanity.com/paper/2504.11879/) (2025)
- [AsymLoc: Towards Asymmetric Feature Matching for Efficient Visual Localization](https://mlsanity.com/paper/2604.09445/) (2026)

## Abstract (arXiv)

We tackle the problem of visual search under resource constraints. Existing systems use the same embedding model to compute representations (embeddings) for the query and gallery images. Such systems inherently face a hard accuracy-efficiency trade-off: the embedding model needs to be large enough to ensure high accuracy, yet small enough to enable query-embedding computation on resource-constrained platforms. This trade-off could be mitigated if gallery embeddings are generated from a large model and query embeddings are extracted using a compact model. The key to building such a system is to ensure representation compatibility between the query and gallery models. In this paper, we address two forms of compatibility: One enforced by modifying the parameters of each model that computes the embeddings. The other by modifying the architectures that compute the embeddings, leading to compatibility-aware neural architecture search (CMP-NAS). We test CMP-NAS on challenging retrieval tasks for fashion images (DeepFashion2), and face images (IJB-C). Compared to ordinary (homogeneous) visual search using the largest embedding model (paragon), CMP-NAS achieves 80-fold and 23-fold cost reduction while maintaining accuracy within 0.3% and 1.6% of the paragon on DeepFashion2 and IJB-C respectively.