# Rethinking Music Captioning with Music Metadata LLMs

arXiv: https://arxiv.org/abs/2602.03023 · HTML: https://mlsanity.com/paper/2602.03023/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models](https://mlsanity.com/paper/2507.08128/) (2025)
- [Listen, Think, and Understand](https://mlsanity.com/paper/2305.10790/) (2023)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [LP-MusicCaps: LLM-Based Pseudo Music Captioning](https://mlsanity.com/paper/2307.16372/) (2023)
- [Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning](https://mlsanity.com/paper/2308.11276/) (2023)
- [ChatMusician: Understanding and Generating Music Intrinsically with LLM](https://mlsanity.com/paper/2402.16153/) (2024)
- [MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response](https://mlsanity.com/paper/2309.08730/) (2023)
- [LLark: A Multimodal Instruction-Following Language Model for Music](https://mlsanity.com/paper/2310.07160/) (2023)
- [M2UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models](https://mlsanity.com/paper/2311.11255/) (2023)
- [MusCaps: Generating Captions for Music Audio](https://mlsanity.com/paper/2104.11984/) (2021)
- [Language-Guided Music Recommendation for Video via Prompt Analogies](https://mlsanity.com/paper/2306.09327/) (2023)
- [Transforming Visual Scene Graphs to Image Captions](https://mlsanity.com/paper/2305.02177/) (2023)
- [JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadata](https://mlsanity.com/paper/2502.07461/) (2025)
- [OpenMU: Your Swiss Army Knife for Music Understanding](https://mlsanity.com/paper/2410.15573/) (2024)
- [Enriching Music Descriptions with a Finetuned-LLM and Metadata for Text-to-Music Retrieval](https://mlsanity.com/paper/2410.03264/) (2024)
- [Re-Bottleneck: Latent Re-Structuring for Neural Audio Autoencoders](https://mlsanity.com/paper/2507.07867/) (2025)
- [Augment, Drop & Swap: Improving Diversity in LLM Captions for Efficient Music-Text Representation Learning](https://mlsanity.com/paper/2409.11498/) (2024)

## Built on by (most cited first, 2 of 2)

- [Music Playlist Captioning at Scale with Large Language Models](https://mlsanity.com/paper/2606.22460/) (2026)
- [AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Models](https://mlsanity.com/paper/2606.14591/) (2026)

## Abstract (arXiv)

Music captioning, or the task of generating a natural language description of music, is useful for both music understanding and controllable music generation. Training captioning models, however, typically requires high-quality music caption data which is scarce compared to metadata (e.g., genre, mood, etc.). As a result, it is common to use large language models (LLMs) to synthesize captions from metadata to generate training data for captioning models, though this process imposes a fixed stylization and entangles factual information with natural language style. As a more direct approach, we propose metadata-based captioning. We train a metadata prediction model to infer detailed music metadata from audio and then convert it into expressive captions via pre-trained LLMs at inference time. Compared to a strong end-to-end baseline trained on LLM-generated captions derived from metadata, our method: (1) achieves comparable performance in less training time over end-to-end captioners, (2) offers flexibility to easily change stylization post-training, enabling output captions to be tailored to specific stylistic and quality requirements, and (3) can be prompted with audio and partial metadata to enable powerful metadata imputation or in-filling--a common task for organizing music data.