# Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval

arXiv: https://arxiv.org/abs/2007.12163 · HTML: https://mlsanity.com/paper/2007.12163/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2020.

## Built on by (most cited first, 30 of 98)

- [LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant](https://mlsanity.com/paper/2412.01720/) (2024)
- [Cross Modal Retrieval with Querybank Normalisation](https://mlsanity.com/paper/2112.12777/) (2021)
- [Deep Learning for Instance Retrieval: A Survey](https://mlsanity.com/paper/2101.11282/) (2021)
- [Recall@k Surrogate Loss with Large Batches and Similarity Mixup](https://mlsanity.com/paper/2108.11179/) (2021)
- [Large-Scale Spatio-Temporal Person Re-identification: Algorithms and Benchmark](https://mlsanity.com/paper/2105.15076/) (2021)
- [GeneCIS: A Benchmark for General Conditional Image Similarity](https://mlsanity.com/paper/2306.07969/) (2023)
- [On Semantic Similarity in Video Retrieval](https://mlsanity.com/paper/2103.10095/) (2021)
- [Improving Point Cloud Based Place Recognition with Ranking-based Loss and Large Batch Training](https://mlsanity.com/paper/2203.00972/) (2022)
- [Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence](https://mlsanity.com/paper/2104.08736/) (2021)
- [Towards Universal Image Embeddings: A Large-Scale Dataset and Challenge for Generic Image Representations](https://mlsanity.com/paper/2309.01858/) (2023)
- [Face, Body, Voice: Video Person-Clustering with Multiple Modalities](https://mlsanity.com/paper/2105.09939/) (2021)
- [Noise-resistant Deep Metric Learning with Ranking-based Instance Selection](https://mlsanity.com/paper/2103.16047/) (2021)
- [Robust and Decomposable Average Precision for Image Retrieval](https://mlsanity.com/paper/2110.01445/) (2021)
- [Evaluation-oriented Knowledge Distillation for Deep Face Recognition](https://mlsanity.com/paper/2206.02325/) (2022)
- [Hierarchical Proxy-based Loss for Deep Metric Learning](https://mlsanity.com/paper/2103.13538/) (2021)
- [Learning Hierarchical Graph Neural Networks for Image Clustering](https://mlsanity.com/paper/2107.01319/) (2021)
- [LibAUC: A Deep Learning Library for X-Risk Optimization](https://mlsanity.com/paper/2306.03065/) (2023)
- [Dynamic Weighted Combiner for Mixed-Modal Image Retrieval](https://mlsanity.com/paper/2312.06179/) (2023)
- [Momentum Accelerates the Convergence of Stochastic AUPRC Maximization](https://mlsanity.com/paper/2107.01173/) (2021)
- [Hierarchical classification at multiple operating points](https://mlsanity.com/paper/2210.10929/) (2022)
- [Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuning](https://mlsanity.com/paper/2411.19458/) (2024)
- [Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control](https://mlsanity.com/paper/2402.17535/) (2024)
- [Diagnosing Human-object Interaction Detectors](https://mlsanity.com/paper/2308.08529/) (2023)
- [TransRank: Self-supervised Video Representation Learning via Ranking-based Transformation Recognition](https://mlsanity.com/paper/2205.02028/) (2022)
- [Optimization of Rank Losses for Image Retrieval](https://mlsanity.com/paper/2309.08250/) (2023)
- [How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?](https://mlsanity.com/paper/2403.07203/) (2024)
- [Automated Video Labelling: Identifying Faces by Corroborative Evidence](https://mlsanity.com/paper/2102.05645/) (2021)
- [MGH: Metadata Guided Hypergraph Modeling for Unsupervised Person Re-identification](https://mlsanity.com/paper/2110.05886/) (2021)
- [Privacy-Preserving Model Upgrades with Bidirectional Compatible Training in Image Retrieval](https://mlsanity.com/paper/2204.13919/) (2022)
- [Not All Pairs are Equal: Hierarchical Learning for Average-Precision-Oriented Video Retrieval](https://mlsanity.com/paper/2407.15566/) (2024)

## Abstract (arXiv)

Optimising a ranking-based metric, such as Average Precision (AP), is notoriously challenging due to the fact that it is non-differentiable, and hence cannot be optimised directly using gradient-descent methods. To this end, we introduce an objective that optimises instead a smoothed approximation of AP, coined Smooth-AP. Smooth-AP is a plug-and-play objective function that allows for end-to-end training of deep networks with a simple and elegant implementation. We also present an analysis for why directly optimising the ranking based metric of AP offers benefits over other deep metric learning losses. We apply Smooth-AP to standard retrieval benchmarks: Stanford Online products and VehicleID, and also evaluate on larger-scale datasets: INaturalist for fine-grained category retrieval, and VGGFace2 and IJB-C for face retrieval. In all cases, we improve the performance over the state-of-the-art, especially for larger-scale datasets, thus demonstrating the effectiveness and scalability of Smooth-AP to real-world scenarios.