# WikiMuTe: A web-sourced dataset of semantic descriptions for music audio

arXiv: https://arxiv.org/abs/2312.09207 · HTML: https://mlsanity.com/paper/2312.09207/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016)
- [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023)
- [ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data](https://mlsanity.com/paper/2001.07966/) (2020)
- [MuLan: A Joint Embedding of Music Audio and Natural Language](https://mlsanity.com/paper/2208.12415/) (2022)
- [LP-MusicCaps: LLM-Based Pseudo Music Captioning](https://mlsanity.com/paper/2307.16372/) (2023)
- [Contrastive Audio-Language Learning for Music](https://mlsanity.com/paper/2208.12208/) (2022)
- [Semi-Supervised Music Tagging Transformer](https://mlsanity.com/paper/2111.13457/) (2021)
- [Toward Universal Text-to-Music Retrieval](https://mlsanity.com/paper/2211.14558/) (2022)
- [Language-Guided Music Recommendation for Video via Prompt Analogies](https://mlsanity.com/paper/2306.09327/) (2023)
- [Multimodal Metric Learning for Tag-based Music Retrieval](https://mlsanity.com/paper/2010.16030/) (2020)
- [Learning music audio representations via weak language supervision](https://mlsanity.com/paper/2112.04214/) (2021)
- [The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future use](https://mlsanity.com/paper/1306.1461/) (2013)
- [Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research](https://mlsanity.com/paper/2110.08975/) (2021)

## Built on by (most cited first, 3 of 3)

- [Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer](https://mlsanity.com/paper/2410.05151/) (2024)
- [MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners](https://mlsanity.com/paper/2506.18729/) (2025)
- [TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control](https://mlsanity.com/paper/2603.09332/) (2026)

## Abstract (arXiv)

Multi-modal deep learning techniques for matching free-form text with music have shown promising results in the field of Music Information Retrieval (MIR). Prior work is often based on large proprietary data while publicly available datasets are few and small in size. In this study, we present WikiMuTe, a new and open dataset containing rich semantic descriptions of music. The data is sourced from Wikipedia's rich catalogue of articles covering musical works. Using a dedicated text-mining pipeline, we extract both long and short-form descriptions covering a wide range of topics related to music content such as genre, style, mood, instrumentation, and tempo. To show the use of this data, we train a model that jointly learns text and audio representations and performs cross-modal retrieval. The model is evaluated on two tasks: tag-based music retrieval and music auto-tagging. The results show that while our approach has state-of-the-art performance on multiple tasks, but still observe a difference in performance depending on the data used for training.