# Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning

arXiv: https://arxiv.org/abs/1904.06627 · HTML: https://mlsanity.com/paper/1904.06627/

## Key facts

- Cited by 369 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2019.

## Built on by (most cited first, 30 of 369)

- [ABO: Dataset and Benchmarks for Real-World 3D Object Understanding](https://mlsanity.com/paper/2110.06199/) (2021)
- [MixVPR: Feature Mixing for Visual Place Recognition](https://mlsanity.com/paper/2303.02190/) (2023)
- [Optimal Transport Aggregation for Visual Place Recognition](https://mlsanity.com/paper/2311.15937/) (2023)
- [EigenPlaces: Training Viewpoint Robust Models for Visual Place Recognition](https://mlsanity.com/paper/2308.10832/) (2023)
- [Hyperbolic Vision Transformers: Combining Improvements in Metric Learning](https://mlsanity.com/paper/2203.10833/) (2022)
- [GSV-Cities: Toward Appropriate Supervised Visual Place Recognition](https://mlsanity.com/paper/2210.10239/) (2022)
- [Training Vision Transformers for Image Retrieval](https://mlsanity.com/paper/2102.05644/) (2021)
- [Cross Modal Retrieval with Querybank Normalisation](https://mlsanity.com/paper/2112.12777/) (2021)
- [Deep Learning for Instance Retrieval: A Survey](https://mlsanity.com/paper/2101.11282/) (2021)
- [BoQ: A Place is Worth a Bag of Learnable Queries](https://mlsanity.com/paper/2405.07364/) (2024)
- [MegaLoc: One Retrieval to Place Them All](https://mlsanity.com/paper/2502.17237/) (2025)
- [CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition](https://mlsanity.com/paper/2402.19231/) (2024)
- [Understanding and Mitigating Annotation Bias in Facial Expression Recognition](https://mlsanity.com/paper/2108.08504/) (2021)
- [Recall@k Surrogate Loss with Large Batches and Similarity Mixup](https://mlsanity.com/paper/2108.11179/) (2021)
- [Improving Point Cloud Based Place Recognition with Ranking-based Loss and Large Batch Training](https://mlsanity.com/paper/2203.00972/) (2022)
- [Data-efficient Large Scale Place Recognition with Graded Similarity Supervision](https://mlsanity.com/paper/2303.11739/) (2023)
- [von Mises-Fisher Loss: An Exploration of Embedding Geometries for Supervised Learning](https://mlsanity.com/paper/2103.15718/) (2021)
- [SphereFace2: Binary Classification is All You Need for Deep Face Recognition](https://mlsanity.com/paper/2108.01513/) (2021)
- [Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistency](https://mlsanity.com/paper/2306.02109/) (2023)
- [Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning](https://mlsanity.com/paper/2211.12878/) (2022)
- [Unicom: Universal and Compact Representation Learning for Image Retrieval](https://mlsanity.com/paper/2304.05884/) (2023)
- [Learning Intra-Batch Connections for Deep Metric Learning](https://mlsanity.com/paper/2102.07753/) (2021)
- [Non-isotropy Regularization for Proxy-based Deep Metric Learning](https://mlsanity.com/paper/2203.08547/) (2022)
- [Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization](https://mlsanity.com/paper/2108.05449/) (2021)
- [Towards Interpretable Deep Metric Learning with Structural Matching](https://mlsanity.com/paper/2108.05889/) (2021)
- [Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking](https://mlsanity.com/paper/2105.14398/) (2021)
- [Close, But Not There: Boosting Geographic Distance Sensitivity in Visual Place Recognition](https://mlsanity.com/paper/2407.02422/) (2024)
- [HIER: Metric Learning Beyond Class Labels via Hierarchical Regularization](https://mlsanity.com/paper/2212.14258/) (2022)
- [ConvFiT: Conversational Fine-Tuning of Pretrained Language Models](https://mlsanity.com/paper/2109.10126/) (2021)
- [Noise-resistant Deep Metric Learning with Ranking-based Instance Selection](https://mlsanity.com/paper/2103.16047/) (2021)

## Abstract (arXiv)

A family of loss functions built on pair-based computation have been proposed in the literature which provide a myriad of solutions for deep metric learning. In this paper, we provide a general weighting framework for understanding recent pair-based loss functions. Our contributions are three-fold: (1) we establish a General Pair Weighting (GPW) framework, which casts the sampling problem of deep metric learning into a unified view of pair weighting through gradient analysis, providing a powerful tool for understanding recent pair-based loss functions; (2) we show that with GPW, various existing pair-based methods can be compared and discussed comprehensively, with clear differences and key limitations identified; (3) we propose a new loss called multi-similarity loss (MS loss) under the GPW, which is implemented in two iterative steps (i.e., mining and weighting). This allows it to fully consider three similarities for pair weighting, providing a more principled approach for collecting and weighting informative pairs. Finally, the proposed MS loss obtains new state-of-the-art performance on four image retrieval benchmarks, where it outperforms the most recent approaches, such as ABEciteKim2018ECCV and HTL by a large margin: 60.6% to 65.7% on CUB200, and 80.9% to 88.0% on In-Shop Clothes Retrieval dataset at Recall@1. Code is available at https://github.com/MalongTech/research-ms-loss.