# Complete Cross-triplet Loss in Label Space for Audio-visual Cross-modal Retrieval

arXiv: https://arxiv.org/abs/2211.03434 · HTML: https://mlsanity.com/paper/2211.03434/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 7 Nov 2022.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [Dual-Path Convolutional Image-Text Embeddings with Instance Loss](https://mlsanity.com/paper/1711.05535/) (2017)
- [Deep Cross-Modal Audio-Visual Generation](https://mlsanity.com/paper/1704.08292/) (2017)
- [Deep Cross-Modal Hashing](https://mlsanity.com/paper/1602.02255/) (2016)
- [Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval](https://mlsanity.com/paper/1908.03737/) (2019)
- [Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA](https://mlsanity.com/paper/1908.03744/) (2019)
- [Unsupervised Generative Adversarial Cross-modal Hashing](https://mlsanity.com/paper/1712.00358/) (2017)
- [Learning Audio-Visual Correlations from Variational Cross-Modal Generation](https://mlsanity.com/paper/2102.03424/) (2021)
- [Category-Based Deep CCA for Fine-Grained Venue Discovery from Multimodal Data](https://mlsanity.com/paper/1805.02997/) (2018)
- [End-to-End Cross-Modality Retrieval with CCA Projections and Pairwise Ranking Loss](https://mlsanity.com/paper/1705.06979/) (2017)
- [TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation](https://mlsanity.com/paper/2110.13412/) (2021)
- [Learning Joint Embedding for Cross-Modal Retrieval](https://mlsanity.com/paper/1908.07673/) (2019)
- [Unsupervised Generative Adversarial Alignment Representation for Sheet music, Audio and Lyrics](https://mlsanity.com/paper/2007.14856/) (2020)

## Built on by (most cited first, 6 of 6)

- [Connecting Multi-modal Contrastive Representations](https://mlsanity.com/paper/2305.14381/) (2023)
- [VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning](https://mlsanity.com/paper/2309.15494/) (2023)
- [Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval](https://mlsanity.com/paper/2310.13451/) (2023)
- [Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning](https://mlsanity.com/paper/2501.09608/) (2025)
- [Learning Audio-Visual Embeddings with Inferred Latent Interaction Graphs](https://mlsanity.com/paper/2601.11995/) (2026)
- [Variance & Greediness: A comparative study of metric-learning losses](https://mlsanity.com/paper/2601.21450/) (2026)

## Abstract (arXiv)

The heterogeneity gap problem is the main challenge in cross-modal retrieval. Because cross-modal data (e.g. audiovisual) have different distributions and representations that cannot be directly compared. To bridge the gap between audiovisual modalities, we learn a common subspace for them by utilizing the intrinsic correlation in the natural synchronization of audio-visual data with the aid of annotated labels. TNN-CCCA is the best audio-visual cross-modal retrieval (AV-CMR) model so far, but the model training is sensitive to hard negative samples when learning common subspace by applying triplet loss to predict the relative distance between inputs. In this paper, to reduce the interference of hard negative samples in representation learning, we propose a new AV-CMR model to optimize semantic features by directly predicting labels and then measuring the intrinsic correlation between audio-visual data using complete cross-triple loss. In particular, our model projects audio-visual features into label space by minimizing the distance between predicted label features after feature projection and ground label representations. Moreover, we adopt complete cross-triplet loss to optimize the predicted label features by leveraging the relationship between all possible similarity and dissimilarity semantic information across modalities. The extensive experimental results on two audio-visual double-checked datasets have shown an improvement of approximately 2.1% in terms of average MAP over the current state-of-the-art method TNN-CCCA for the AV-CMR task, which indicates the effectiveness of our proposed model.