# Learning Audio Embeddings with User Listening Data for Content-based Music Recommendation

arXiv: https://arxiv.org/abs/2010.15389 · HTML: https://mlsanity.com/paper/2010.15389/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2020.

## Built on by (most cited first, 6 of 6)

- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Data](https://mlsanity.com/paper/2112.07891/) (2021)
- [TONet: Tone-Octave Network for Singing Melody Extraction from Polyphonic Music](https://mlsanity.com/paper/2202.00951/) (2022)
- [Towards Improving Harmonic Sensitivity and Prediction Stability for Singing Melody Extraction](https://mlsanity.com/paper/2308.02723/) (2023)
- [Prompt Tuning for Item Cold-start Recommendation](https://mlsanity.com/paper/2412.18082/) (2024)
- [Evaluating Multimodal Large Language Models on Core Music Perception Tasks](https://mlsanity.com/paper/2510.22455/) (2025)

## Abstract (arXiv)

Personalized recommendation on new track releases has always been a challenging problem in the music industry. To combat this problem, we first explore user listening history and demographics to construct a user embedding representing the user's music preference. With the user embedding and audio data from user's liked and disliked tracks, an audio embedding can be obtained for each track using metric learning with Siamese networks. For a new track, we can decide the best group of users to recommend by computing the similarity between the track's audio embedding and different user embeddings, respectively. The proposed system yields state-of-the-art performance on content-based music recommendation tested with millions of users and tracks. Also, we extract audio embeddings as features for music genre classification tasks. The results show the generalization ability of our audio embeddings.