# Anchor-aware Deep Metric Learning for Audio-visual Retrieval

arXiv: https://arxiv.org/abs/2404.13789 · HTML: https://mlsanity.com/paper/2404.13789/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 21 Apr 2024.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Dual-Path Convolutional Image-Text Embeddings with Instance Loss](https://mlsanity.com/paper/1711.05535/) (2017)
- [Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss](https://mlsanity.com/paper/2109.04290/) (2021)
- [Hyperbolic Vision Transformers: Combining Improvements in Metric Learning](https://mlsanity.com/paper/2203.10833/) (2022)
- [Visual to Sound: Generating Natural Sound for Videos in the Wild](https://mlsanity.com/paper/1712.01393/) (2017)
- [CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations](https://mlsanity.com/paper/2109.14910/) (2021)
- [Hardness-Aware Deep Metric Learning](https://mlsanity.com/paper/1903.05503/) (2019)
- [HyperML: A Boosting Metric Learning Approach in Hyperbolic Space for Recommender Systems](https://mlsanity.com/paper/1809.01703/) (2018)
- [Embedding Expansion: Augmentation in Embedding Space for Deep Metric Learning](https://mlsanity.com/paper/2003.02546/) (2020)
- [HIER: Metric Learning Beyond Class Labels via Hierarchical Regularization](https://mlsanity.com/paper/2212.14258/) (2022)
- [It Takes Two to Tango: Mixup for Deep Metric Learning](https://mlsanity.com/paper/2106.04990/) (2021)
- [Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval](https://mlsanity.com/paper/1908.03737/) (2019)
- [Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA](https://mlsanity.com/paper/1908.03744/) (2019)
- [Category-Based Deep CCA for Fine-Grained Venue Discovery from Multimodal Data](https://mlsanity.com/paper/1805.02997/) (2018)
- [CNN-based Patch Matching for Optical Flow with Thresholded Hinge Embedding Loss](https://mlsanity.com/paper/1607.08064/) (2016)
- [Variational Autoencoder with CCA for Audio-Visual Cross-Modal Retrieval](https://mlsanity.com/paper/2112.02601/) (2021)
- [VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning](https://mlsanity.com/paper/2309.15494/) (2023)
- [Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval](https://mlsanity.com/paper/2310.13451/) (2023)

## Built on by (most cited first, 4 of 4)

- [Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning](https://mlsanity.com/paper/2501.09608/) (2025)
- [Learning Audio-Visual Embeddings with Inferred Latent Interaction Graphs](https://mlsanity.com/paper/2601.11995/) (2026)
- [Multimodal Fusion via Self-Consistent Task-Gradient Fields](https://mlsanity.com/paper/2410.15475/) (2024)
- [Variance & Greediness: A comparative study of metric-learning losses](https://mlsanity.com/paper/2601.21450/) (2026)

## Abstract (arXiv)

Metric learning minimizes the gap between similar (positive) pairs of data points and increases the separation of dissimilar (negative) pairs, aiming at capturing the underlying data structure and enhancing the performance of tasks like audio-visual cross-modal retrieval (AV-CMR). Recent works employ sampling methods to select impactful data points from the embedding space during training. However, the model training fails to fully explore the space due to the scarcity of training data points, resulting in an incomplete representation of the overall positive and negative distributions. In this paper, we propose an innovative Anchor-aware Deep Metric Learning (AADML) method to address this challenge by uncovering the underlying correlations among existing data points, which enhances the quality of the shared embedding space. Specifically, our method establishes a correlation graph-based manifold structure by considering the dependencies between each sample as the anchor and its semantically similar samples. Through dynamic weighting of the correlations within this underlying manifold structure using an attention-driven mechanism, Anchor Awareness (AA) scores are obtained for each anchor. These AA scores serve as data proxies to compute relative distances in metric learning approaches. Extensive experiments conducted on two audio-visual benchmark datasets demonstrate the effectiveness of our proposed AADML method, significantly surpassing state-of-the-art models. Furthermore, we investigate the integration of AA proxies with various metric learning methods, further highlighting the efficacy of our approach.