# Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval

arXiv: https://arxiv.org/abs/2310.13451 · HTML: https://mlsanity.com/paper/2310.13451/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition](https://mlsanity.com/paper/2004.00288/) (2020)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Dual-Path Convolutional Image-Text Embeddings with Instance Loss](https://mlsanity.com/paper/1711.05535/) (2017)
- [Visual to Sound: Generating Natural Sound for Videos in the Wild](https://mlsanity.com/paper/1712.01393/) (2017)
- [Competence-based Multimodal Curriculum Learning for Medical Report Generation](https://mlsanity.com/paper/2206.14579/) (2022)
- [Embedding Expansion: Augmentation in Embedding Space for Deep Metric Learning](https://mlsanity.com/paper/2003.02546/) (2020)
- [Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval](https://mlsanity.com/paper/1908.03737/) (2019)
- [Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA](https://mlsanity.com/paper/1908.03744/) (2019)
- [Category-Based Deep CCA for Fine-Grained Venue Discovery from Multimodal Data](https://mlsanity.com/paper/1805.02997/) (2018)
- [Complete Cross-triplet Loss in Label Space for Audio-visual Cross-modal Retrieval](https://mlsanity.com/paper/2211.03434/) (2022)
- [VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning](https://mlsanity.com/paper/2309.15494/) (2023)
- [MusicTM-Dataset for Joint Representation Learning among Sheet Music, Lyrics, and Musical Audio](https://mlsanity.com/paper/2012.00290/) (2020)

## Built on by (most cited first, 4 of 4)

- [Anchor-aware Deep Metric Learning for Audio-visual Retrieval](https://mlsanity.com/paper/2404.13789/) (2024)
- [Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning](https://mlsanity.com/paper/2501.09608/) (2025)
- [Learning Audio-Visual Embeddings with Inferred Latent Interaction Graphs](https://mlsanity.com/paper/2601.11995/) (2026)
- [Variance & Greediness: A comparative study of metric-learning losses](https://mlsanity.com/paper/2601.21450/) (2026)

## Abstract (arXiv)

The cross-modal retrieval model leverages the potential of triple loss optimization to learn robust embedding spaces. However, existing methods often train these models in a singular pass, overlooking the distinction between semi-hard and hard triples in the optimization process. The oversight of not distinguishing between semi-hard and hard triples leads to suboptimal model performance. In this paper, we introduce a novel approach rooted in curriculum learning to address this problem. We propose a two-stage training paradigm that guides the model's learning process from semi-hard to hard triplets. In the first stage, the model is trained with a set of semi-hard triplets, starting from a low-loss base. Subsequently, in the second stage, we augment the embeddings using an interpolation technique. This process identifies potential hard negatives, alleviating issues arising from high-loss functions due to a scarcity of hard triples. Our approach then applies hard triplet mining in the augmented embedding space to further optimize the model. Extensive experimental results conducted on two audio-visual datasets show a significant improvement of approximately 9.8% in terms of average Mean Average Precision (MAP) over the current state-of-the-art method, MSNSCA, for the Audio-Visual Cross-Modal Retrieval (AV-CMR) task on the AVE dataset, indicating the effectiveness of our proposed method.