# Learning Music Sequence Representation from Text Supervision

arXiv: https://arxiv.org/abs/2305.19602 · HTML: https://mlsanity.com/paper/2305.19602/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2103.09410/) (2021)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [musicnn: Pre-trained convolutional neural networks for music audio tagging](https://mlsanity.com/paper/1909.06654/) (2019)
- [A Simple Long-Tailed Recognition Baseline via Vision-Language Model](https://mlsanity.com/paper/2111.14745/) (2021)
- [Automatic tagging using deep convolutional neural networks](https://mlsanity.com/paper/1606.00298/) (2016)
- [Toward Interpretable Music Tagging with Self-Attention](https://mlsanity.com/paper/1906.04972/) (2019)
- [End-to-end learning for music audio tagging at scale](https://mlsanity.com/paper/1711.02520/) (2017)
- [ESResNe(X)t-fbsp: Learning Robust Time-Frequency Transformation of Audio](https://mlsanity.com/paper/2104.11587/) (2021)
- [Deep Learning and Music Adversaries](https://mlsanity.com/paper/1507.04761/) (2015)
- [Music Genre Classification using Machine Learning Techniques](https://mlsanity.com/paper/1804.01149/) (2018)
- [Representation Learning of Music Using Artist, Album, and Track Information](https://mlsanity.com/paper/1906.11783/) (2019)

## Built on by (most cited first, 5 of 5)

- [Heterogeneous Contrastive Learning for Foundation Models and Beyond](https://mlsanity.com/paper/2404.00225/) (2024)
- [Text-to-Song: Towards Controllable Music Generation Incorporating Vocals and Accompaniment](https://mlsanity.com/paper/2404.09313/) (2024)
- [Augment, Drop & Swap: Improving Diversity in LLM Captions for Efficient Music-Text Representation Learning](https://mlsanity.com/paper/2409.11498/) (2024)
- [Unsupervised Musical Object Discovery from Audio](https://mlsanity.com/paper/2311.07534/) (2023)
- [MDSC: Towards Evaluating the Style Consistency Between Music and Dance](https://mlsanity.com/paper/2309.01340/) (2023)

## Abstract (arXiv)

Music representation learning is notoriously difficult for its complex human-related concepts contained in the sequence of numerical signals. To excavate better MUsic SEquence Representation from labeled audio, we propose a novel text-supervision pre-training method, namely MUSER. MUSER adopts an audio-spectrum-text tri-modal contrastive learning framework, where the text input could be any form of meta-data with the help of text templates while the spectrum is derived from an audio sequence. Our experiments reveal that MUSER could be more flexibly adapted to downstream tasks compared with the current data-hungry pre-training method, and it only requires 0.056% of pre-training data to achieve the state-of-the-art performance.