# Multimodal Metric Learning for Tag-based Music Retrieval

arXiv: https://arxiv.org/abs/2010.16030 · HTML: https://mlsanity.com/paper/2010.16030/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2020.

## Built on by (most cited first, 14 of 14)

- [Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data](https://mlsanity.com/paper/2302.06232/) (2023)
- [Supervised and Unsupervised Learning of Audio Representations for Music Understanding](https://mlsanity.com/paper/2210.03799/) (2022)
- [Contrastive Audio-Language Learning for Music](https://mlsanity.com/paper/2208.12208/) (2022)
- [LLark: A Multimodal Instruction-Following Language Model for Music](https://mlsanity.com/paper/2310.07160/) (2023)
- [SpecTNT: a Time-Frequency Transformer for Music Audio](https://mlsanity.com/paper/2110.09127/) (2021)
- [Safeguarding Data in Multimodal AI: A Differentially Private Approach to CLIP Training](https://mlsanity.com/paper/2306.08173/) (2023)
- [Deep Learning Based EDM Subgenre Classification using Mel-Spectrogram and Tempogram Features](https://mlsanity.com/paper/2110.08862/) (2021)
- [Enhancing Affective Representations of Music-Induced EEG through Multimodal Supervision and latent Domain Adaptation](https://mlsanity.com/paper/2202.09750/) (2022)
- [Learning Unsupervised Hierarchies of Audio Concepts](https://mlsanity.com/paper/2207.11231/) (2022)
- [WikiMuTe: A web-sourced dataset of semantic descriptions for music audio](https://mlsanity.com/paper/2312.09207/) (2023)
- [DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuning](https://mlsanity.com/paper/2502.12623/) (2025)
- [Sonus Texere! Automated Dense Soundtrack Construction for Books using Movie Adaptations](https://mlsanity.com/paper/2212.01033/) (2022)
- [MDSC: Towards Evaluating the Style Consistency Between Music and Dance](https://mlsanity.com/paper/2309.01340/) (2023)
- [Solving Copyright Infringement on Short Video Platforms: Novel Datasets and an Audio Restoration Deep Learning Pipeline](https://mlsanity.com/paper/2504.21772/) (2025)

## Abstract (arXiv)

Tag-based music retrieval is crucial to browse large-scale music libraries efficiently. Hence, automatic music tagging has been actively explored, mostly as a classification task, which has an inherent limitation: a fixed vocabulary. On the other hand, metric learning enables flexible vocabularies by using pretrained word embeddings as side information. Also, metric learning has already proven its suitability for cross-modal retrieval tasks in other domains (e.g., text-to-image) by jointly learning a multimodal embedding space. In this paper, we investigate three ideas to successfully introduce multimodal metric learning for tag-based music retrieval: elaborate triplet sampling, acoustic and cultural music information, and domain-specific word embeddings. Our experimental results show that the proposed ideas enhance the retrieval system quantitatively, and qualitatively. Furthermore, we release the MSD500, a subset of the Million Song Dataset (MSD) containing 500 cleaned tags, 7 manually annotated tag categories, and user taste profiles.