# Cross-modal Embeddings for Video and Audio Retrieval

arXiv: https://arxiv.org/abs/1801.02200 · HTML: https://mlsanity.com/paper/1801.02200/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2018.

## Built on by (most cited first, 21 of 21)

- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images](https://mlsanity.com/paper/2306.07257/) (2023)
- [Sound-Guided Semantic Image Manipulation](https://mlsanity.com/paper/2112.00007/) (2021)
- [Noisy Correspondence Learning with Meta Similarity Correction](https://mlsanity.com/paper/2304.06275/) (2023)
- [TVLT: Textless Vision-Language Transformer](https://mlsanity.com/paper/2209.14156/) (2022)
- [It's Time for Artistic Correspondence in Music and Video](https://mlsanity.com/paper/2206.07148/) (2022)
- [Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser](https://mlsanity.com/paper/2305.17343/) (2023)
- [A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!](https://mlsanity.com/paper/2206.13390/) (2022)
- [Robust Sound-Guided Image Manipulation](https://mlsanity.com/paper/2208.14114/) (2022)
- [TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation](https://mlsanity.com/paper/2110.13412/) (2021)
- [ImagebindDC: Compressing Multi-modal Data with Imagebind-based Condensation](https://mlsanity.com/paper/2511.08263/) (2025)
- [Music-to-Text Synaesthesia: Generating Descriptive Text from Music Recordings](https://mlsanity.com/paper/2210.00434/) (2022)
- [Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding](https://mlsanity.com/paper/2306.10684/) (2023)
- [Zero-shot Audio Topic Reranking using Large Language Models](https://mlsanity.com/paper/2309.07606/) (2023)
- [Introducing Auxiliary Text Query-modifier to Content-based Audio Retrieval](https://mlsanity.com/paper/2207.09732/) (2022)
- [Video-to-Music Recommendation using Temporal Alignment of Segments](https://mlsanity.com/paper/2306.07187/) (2023)
- [Start from Video-Music Retrieval: An Inter-Intra Modal Loss for Cross Modal Retrieval](https://mlsanity.com/paper/2407.19415/) (2024)
- [DETECLAP: Enhancing Audio-Visual Representation Learning with Object Information](https://mlsanity.com/paper/2409.11729/) (2024)
- [Learning Audio-Visual Embeddings with Inferred Latent Interaction Graphs](https://mlsanity.com/paper/2601.11995/) (2026)
- [Semantic Video Moments Retrieval at Scale: A New Task and a Baseline](https://mlsanity.com/paper/2210.08389/) (2022)
- [VMCML: Video and Music Matching via Cross-Modality Lifting](https://mlsanity.com/paper/2303.12379/) (2023)

## Abstract (arXiv)

The increasing amount of online videos brings several opportunities for training self-supervised neural networks. The creation of large scale datasets of videos such as the YouTube-8M allows us to deal with this large amount of data in manageable way. In this work, we find new ways of exploiting this dataset by taking advantage of the multi-modal information it provides. By means of a neural network, we are able to create links between audio and visual documents, by projecting them into a common region of the feature space, obtaining joint audio-visual embeddings. These links are used to retrieve audio samples that fit well to a given silent video, and also to retrieve images that match a given a query audio. The results in terms of Recall@K obtained over a subset of YouTube-8M videos show the potential of this unsupervised approach for cross-modal feature learning. We train embeddings for both scales and assess their quality in a retrieval problem, formulated as using the feature extracted from one modality to retrieve the most similar videos based on the features computed in the other modality.