# MusCaps: Generating Captions for Music Audio

arXiv: https://arxiv.org/abs/2104.11984 · HTML: https://mlsanity.com/paper/2104.11984/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 24 Apr 2021.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge](https://mlsanity.com/paper/1609.06647/) (2016)
- [Improved Image Captioning via Policy Gradient optimization of SPIDEr](https://mlsanity.com/paper/1612.00370/) (2016)
- [Evaluation of CNN-based Automatic Music Tagging Models](https://mlsanity.com/paper/2006.00751/) (2020)
- [Automatic tagging using deep convolutional neural networks](https://mlsanity.com/paper/1606.00298/) (2016)
- [Automated Audio Captioning with Recurrent Neural Networks](https://mlsanity.com/paper/1706.10006/) (2017)
- [Transfer learning for music classification and regression tasks](https://mlsanity.com/paper/1703.09179/) (2017)
- [Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrieval](https://mlsanity.com/paper/2012.07331/) (2020)
- [End-to-end learning for music audio tagging at scale](https://mlsanity.com/paper/1711.02520/) (2017)
- [A Transformer-based Audio Captioning Model with Keyword Estimation](https://mlsanity.com/paper/2007.00222/) (2020)
- [Audio Caption: Listen and Tell](https://mlsanity.com/paper/1902.09254/) (2019)
- [Video Understanding as Machine Translation](https://mlsanity.com/paper/2006.07203/) (2020)
- [Sample-level CNN Architectures for Music Auto-tagging Using Raw Waveforms](https://mlsanity.com/paper/1710.10451/) (2017)
- [COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations](https://mlsanity.com/paper/2006.08386/) (2020)
- [WaveTransformer: A Novel Architecture for Audio Captioning Based on Learning Temporal and Time-Frequency Information](https://mlsanity.com/paper/2010.11098/) (2020)
- [Towards Music Captioning: Generating Music Playlist Descriptions](https://mlsanity.com/paper/1608.04868/) (2016)
- [Multi-task Regularization Based on Infrequent Classes for Audio Captioning](https://mlsanity.com/paper/2007.04660/) (2020)
- [Audio Captioning using Gated Recurrent Units](https://mlsanity.com/paper/2006.03391/) (2020)
- [Listen carefully and tell: an audio captioning system based on residual learning and gammatone audio representation](https://mlsanity.com/paper/2006.15406/) (2020)

## Built on by (most cited first, 23 of 23)

- [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023)
- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning](https://mlsanity.com/paper/2308.11276/) (2023)
- [The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation](https://mlsanity.com/paper/2311.10057/) (2023)
- [MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response](https://mlsanity.com/paper/2309.08730/) (2023)
- [Contrastive Audio-Language Learning for Music](https://mlsanity.com/paper/2208.12208/) (2022)
- [LLark: A Multimodal Instruction-Following Language Model for Music](https://mlsanity.com/paper/2310.07160/) (2023)
- [MidiCaps: A large-scale MIDI dataset with text captions](https://mlsanity.com/paper/2406.02255/) (2024)
- [MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music](https://mlsanity.com/paper/2402.09871/) (2024)
- [SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing](https://mlsanity.com/paper/2601.09385/) (2026)
- [Interpreting Song Lyrics with an Audio-Informed Pre-trained Language Model](https://mlsanity.com/paper/2208.11671/) (2022)
- [Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation](https://mlsanity.com/paper/2407.20445/) (2024)
- [Vis2Mus: Exploring Multimodal Representation Mapping for Controllable Music Generation](https://mlsanity.com/paper/2211.05543/) (2022)
- [A Survey of Foundation Models for Music Understanding](https://mlsanity.com/paper/2409.09601/) (2024)
- [Music-to-Text Synaesthesia: Generating Descriptive Text from Music Recordings](https://mlsanity.com/paper/2210.00434/) (2022)
- [Rethinking Music Captioning with Music Metadata LLMs](https://mlsanity.com/paper/2602.03023/) (2026)
- [ALCAP: Alignment-Augmented Music Captioner](https://mlsanity.com/paper/2212.10901/) (2022)
- [JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal Parameters Tuning](https://mlsanity.com/paper/2406.12292/) (2024)
- [Prevailing Research Areas for Music AI in the Era of Foundation Models](https://mlsanity.com/paper/2409.09378/) (2024)
- [Serenade: A Model for Human-in-the-loop Automatic Chord Estimation](https://mlsanity.com/paper/2310.11165/) (2023)
- [Combining Genre Classification and Harmonic-Percussive Features with Diffusion Models for Music-Video Generation](https://mlsanity.com/paper/2412.05694/) (2024)
- [SonicVerse: Multi-Task Learning for Music Feature-Informed Captioning](https://mlsanity.com/paper/2506.15154/) (2025)
- [Art2Mus: Artwork-to-Music Generation via Visual Conditioning and Large-Scale Cross-Modal Alignment](https://mlsanity.com/paper/2602.17599/) (2026)

## Abstract (arXiv)

Content-based music information retrieval has seen rapid progress with the adoption of deep learning. Current approaches to high-level music description typically make use of classification models, such as in auto-tagging or genre and mood classification. In this work, we propose to address music description via audio captioning, defined as the task of generating a natural language description of music audio content in a human-like manner. To this end, we present the first music audio captioning model, MusCaps, consisting of an encoder-decoder with temporal attention. Our method combines convolutional and recurrent neural network architectures to jointly process audio-text inputs through a multimodal encoder and leverages pre-training on audio data to obtain representations that effectively capture and summarise musical features in the input. Evaluation of the generated captions through automatic metrics shows that our method outperforms a baseline designed for non-music audio captioning. Through an ablation study, we unveil that this performance boost can be mainly attributed to pre-training of the audio encoder, while other design choices - modality fusion, decoding strategy and the use of attention - contribute only marginally. Our model represents a shift away from classification-based music description and combines tasks requiring both auditory and linguistic understanding to bridge the semantic gap in music information retrieval.