# MidiCaps: A large-scale MIDI dataset with text captions arXiv: https://arxiv.org/abs/2406.02255 ยท HTML: https://mlsanity.com/paper/2406.02255/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Builds on 17 of its 31 references that are in the map. - First posted on arXiv 4 Jun 2024. ## Builds on (most cited first, 17 of 17) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021) - [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022) - [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023) - [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023) - [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023) - [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018) - [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023) - [MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment](https://mlsanity.com/paper/1709.06298/) (2017) - [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023) - [MuLan: A Joint Embedding of Music Audio and Natural Language](https://mlsanity.com/paper/2208.12415/) (2022) - [LP-MusicCaps: LLM-Based Pseudo Music Captioning](https://mlsanity.com/paper/2307.16372/) (2023) - [Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra](https://mlsanity.com/paper/2404.03647/) (2024) - [A Functional Taxonomy of Music Generation Systems](https://mlsanity.com/paper/1812.04186/) (2018) - [MusCaps: Generating Captions for Music Audio](https://mlsanity.com/paper/2104.11984/) (2021) - [Music-to-Text Synaesthesia: Generating Descriptive Text from Music Recordings](https://mlsanity.com/paper/2210.00434/) (2022) ## Built on by (most cited first, 19 of 19) - [Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2505.15957/) (2025) - [Text2midi: Generating Symbolic Music from Captions](https://mlsanity.com/paper/2412.16526/) (2024) - [Frechet Music Distance: A Metric For Generative Symbolic Music Evaluation](https://mlsanity.com/paper/2412.07948/) (2024) - [JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadata](https://mlsanity.com/paper/2502.07461/) (2025) - [CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models](https://mlsanity.com/paper/2410.13267/) (2024) - [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025) - [Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignment](https://mlsanity.com/paper/2505.12669/) (2025) - [TuneJury: An Open Metric for Improving Music Generation Preference Alignment](https://mlsanity.com/paper/2606.17006/) (2026) - [MelodySim: Measuring Melody-aware Music Similarity for Plagiarism Detection](https://mlsanity.com/paper/2505.20979/) (2025) - [Prevailing Research Areas for Music AI in the Era of Foundation Models](https://mlsanity.com/paper/2409.09378/) (2024) - [On the de-duplication of the Lakh MIDI dataset](https://mlsanity.com/paper/2509.16662/) (2025) - [MIDI-LLM: Improving Text-to-MIDI Music Generation via Adapting Large Language Models](https://mlsanity.com/paper/2511.03942/) (2025) - [CompLex: Music Theory Lexicon Constructed by Autonomous Agents for Automatic Music Generation](https://mlsanity.com/paper/2508.19603/) (2025) - [Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approach](https://mlsanity.com/paper/2509.22378/) (2025) - [HNote: Extending YNote with Hexadecimal Encoding for Fine-Tuning LLMs in Music Modeling](https://mlsanity.com/paper/2509.25694/) (2025) - [Decomposer: Learning to Decompile Symbolic Music to Programs](https://mlsanity.com/paper/2607.01849/) (2026) - [MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering](https://mlsanity.com/paper/2608.06638/) (2026) - [MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation](https://mlsanity.com/paper/2608.09035/) (2026) - [MUUNRiver-Bench: Diagnosing Relation-Dependent Music Retrieval with Multimodal Instructions](https://mlsanity.com/paper/2609.16090/) (2026) ## Abstract (arXiv) Generative models guided by text prompts are increasingly becoming more popular. However, no text-to-MIDI models currently exist due to the lack of a captioned MIDI dataset. This work aims to enable research that combines LLMs with symbolic music by presenting, the first openly available large-scale MIDI dataset with text captions. MIDI (Musical Instrument Digital Interface) files are widely used for encoding musical information and can capture the nuances of musical composition. They are widely used by music producers, composers, musicologists, and performers alike. Inspired by recent advancements in captioning techniques, we present a curated dataset of over 168k MIDI files with textual descriptions. Each MIDI caption describes the musical content, including tempo, chord progression, time signature, instruments, genre, and mood, thus facilitating multi-modal exploration and analysis. The dataset encompasses various genres, styles, and complexities, offering a rich data source for training and evaluating models for tasks such as music information retrieval, music understanding, and cross-modal translation. We provide detailed statistics about the dataset and have assessed the quality of the captions in an extensive listening study. We anticipate that this resource will stimulate further research at the intersection of music and natural language processing, fostering advancements in both fields.