# MuseChat: A Conversational Music Recommendation System for Videos

arXiv: https://arxiv.org/abs/2310.06282 · HTML: https://mlsanity.com/paper/2310.06282/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 76 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 30 of 57)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System](https://mlsanity.com/paper/2303.14524/) (2023)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [Advances and Challenges in Conversational Recommender Systems: A Survey](https://mlsanity.com/paper/2101.09459/) (2021)
- [ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models](https://mlsanity.com/paper/2302.07257/) (2023)
- [From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models](https://mlsanity.com/paper/2212.10846/) (2022)
- [Talk-to-Edit: Fine-Grained Facial Editing via Dialog](https://mlsanity.com/paper/2109.04425/) (2021)

## Built on by (most cited first, 21 of 21)

- [Tell What You Hear From What You See -- Video to Audio Generation Through Text](https://mlsanity.com/paper/2411.05679/) (2024)
- [From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation](https://mlsanity.com/paper/2409.19132/) (2024)
- [UniGuard: Towards Universal Safety Guardrails for Jailbreak Attacks on Multimodal Large Language Models](https://mlsanity.com/paper/2411.01703/) (2024)
- [ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction](https://mlsanity.com/paper/2404.15592/) (2024)
- [HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronization](https://mlsanity.com/paper/2503.01725/) (2025)
- [EIVEN: Efficient Implicit Attribute Value Extraction using Multimodal LLM](https://mlsanity.com/paper/2404.08886/) (2024)
- [AutoMV: An Automatic Multi-Agent System for Music Video Generation](https://mlsanity.com/paper/2512.12196/) (2025)
- [Generative Explore-Exploit: Training-free Optimization of Generative Recommender Systems using LLM Optimizers](https://mlsanity.com/paper/2406.05255/) (2024)
- [Hedge Fund Portfolio Construction Using PolyModel Theory and iTransformer](https://mlsanity.com/paper/2408.03320/) (2024)
- [Unveiling and Mitigating Bias in Large Language Model Recommendations: A Path to Fairness](https://mlsanity.com/paper/2409.10825/) (2024)
- [Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation](https://mlsanity.com/paper/2407.20445/) (2024)
- [Mamba Fusion: Learning Actions Through Questioning](https://mlsanity.com/paper/2409.11513/) (2024)
- [FilmComposer: LLM-Driven Music Production for Silent Film Clips](https://mlsanity.com/paper/2503.08147/) (2025)
- [Taming Text-to-Image Synthesis for Novices: User-centric Prompt Generation via Multi-turn Guidance](https://mlsanity.com/paper/2408.12910/) (2024)
- [Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding](https://mlsanity.com/paper/2508.20765/) (2025)
- [CPGRec+: A Balance-oriented Framework for Personalized Video Game Recommendations](https://mlsanity.com/paper/2604.14586/) (2026)
- [Character-based Outfit Generation with Vision-augmented Style Extraction via LLMs](https://mlsanity.com/paper/2402.05941/) (2024)
- [ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction](https://mlsanity.com/paper/2503.20978/) (2025)
- [ChromouVQA: Benchmarking Vision-Language Models under Chromatic Camouflaged Images](https://mlsanity.com/paper/2512.05137/) (2025)
- [DialBGM: A Benchmark for Background Music Recommendation from Everyday Multi-Turn Dialogues](https://mlsanity.com/paper/2604.07895/) (2026)
- [Understanding the Performance Plateau in Text-to-Video Retrieval: A Comprehensive Empirical and Linguistic Analysis](https://mlsanity.com/paper/2605.00826/) (2026)

## Abstract (arXiv)

Music recommendation for videos attracts growing interest in multi-modal research. However, existing systems focus primarily on content compatibility, often ignoring the users' preferences. Their inability to interact with users for further refinements or to provide explanations leads to a less satisfying experience. We address these issues with MuseChat, a first-of-its-kind dialogue-based recommendation system that personalizes music suggestions for videos. Our system consists of two key functionalities with associated modules: recommendation and reasoning. The recommendation module takes a video along with optional information including previous suggested music and user's preference as inputs and retrieves an appropriate music matching the context. The reasoning module, equipped with the power of Large Language Model (Vicuna-7B) and extended to multi-modal inputs, is able to provide reasonable explanation for the recommended music. To evaluate the effectiveness of MuseChat, we build a large-scale dataset, conversational music recommendation for videos, that simulates a two-turn interaction between a user and a recommender based on accurate music track information. Experiment results show that MuseChat achieves significant improvements over existing video-based music retrieval methods as well as offers strong interpretability and interactability.