# Multimodal Music Recommendation System using LLMs

arXiv: https://arxiv.org/abs/2606.00125 · HTML: https://mlsanity.com/paper/2606.00125/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 20 of 20)

- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Neural Graph Collaborative Filtering](https://mlsanity.com/paper/1905.08108/) (2019)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation](https://mlsanity.com/paper/2311.09049/) (2023)
- [Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited](https://mlsanity.com/paper/2303.13835/) (2023)
- [musicnn: Pre-trained convolutional neural networks for music audio tagging](https://mlsanity.com/paper/1909.06654/) (2019)
- [A Foundation Model for Music Informatics](https://mlsanity.com/paper/2311.03318/) (2023)
- [TAGNN: Target Attentive Graph Neural Networks for Session-based Recommendation](https://mlsanity.com/paper/2005.02844/) (2020)
- [MAP-Music2Vec: A Simple and Effective Baseline for Self-Supervised Music Audio Representation Learning](https://mlsanity.com/paper/2212.02508/) (2022)
- [EnCodecMAE: Leveraging neural codecs for universal audio representation learning](https://mlsanity.com/paper/2309.07391/) (2023)
- [Enhancing Sequential Music Recommendation with Personalized Popularity Awareness](https://mlsanity.com/paper/2409.04329/) (2024)
- [X-Reflect: Cross-Reflection Prompting for Multimodal Recommendation](https://mlsanity.com/paper/2408.15172/) (2024)
- [A Deep Multimodal Approach for Cold-start Music Recommendation](https://mlsanity.com/paper/1706.09739/) (2017)
- [Transformers Meet ACT-R: Repeat-Aware and Sequential Listening Session Recommendation](https://mlsanity.com/paper/2408.16578/) (2024)
- [MDVT: Enhancing Multimodal Recommendation with Model-Agnostic Multimodal-Driven Virtual Triplets](https://mlsanity.com/paper/2505.16665/) (2025)
- [Do Recommender Systems Really Leverage Multimodal Content? A Comprehensive Analysis on Multimodal Representations for Recommendation](https://mlsanity.com/paper/2508.04571/) (2025)
- [Multimodal Fusion Based Attentive Networks for Sequential Music Recommendation](https://mlsanity.com/paper/2110.01001/) (2021)
- [Comparative Analysis of Pretrained Audio Representations in Music Recommender Systems](https://mlsanity.com/paper/2409.08987/) (2024)
- [Not Just What, But When: Integrating Irregular Intervals to LLM for Sequential Recommendation](https://mlsanity.com/paper/2507.23209/) (2025)

## Abstract (arXiv)

Music recommendation systems typically treat songs as opaque tokens, relying on collaborative interaction histories which overlooks semantic or acoustic content. Prior work has explored LLM-augmented, multimodal, and text-enhanced approaches to sequential recommendation, and while some methods partially combine semantic, acoustic, or engagement signals, none jointly model all three within a unified LLM-based sequential reasoning framework that grounds recommendations in actual song content. In this work, we propose a multimodal framework for session-based music recommendation that enriches the LastFM-1K dataset with three complementary signals: (1) audio and lyric embeddings extracted using pretrained music and text representation models, (2) LLM-generated semantic metadata using the MGPHot annotation schema, and (3) listening completion ratios. We adopt the E4SRec framework by extending it with multimodal features and different item ID encoder backbones, including SASRec, BERT4Rec, and GRU4Rec. We further extend the LLM backbone option with LLaMa-2-13B, Qwen2.5-7B-Instruct, and LLaMa-3-70B in both zero-shot and fine-tuned settings. Our experiments show that integrating content-based features improves over ID-only baselines up to 95% in terms of Recall and 79% in terms of NDCG. Moreover, our experiments show that naive multimodal fusion does not always yield additive improvements, highlighting challenges in cross-modal integration. We release a large-scale multimodal benchmark for music recommendation.