# Text2midi: Generating Symbolic Music from Captions

arXiv: https://arxiv.org/abs/2412.16526 · HTML: https://mlsanity.com/paper/2412.16526/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 21 Dec 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [DeepBach: a Steerable Model for Bach Chorales Generation](https://mlsanity.com/paper/1612.01010/) (2016)
- [MuseCoco: Generating Symbolic Music from Text](https://mlsanity.com/paper/2306.00110/) (2023)
- [Symphony Generation with Permutation Invariant Language Model](https://mlsanity.com/paper/2205.05448/) (2022)
- [A Functional Taxonomy of Music Generation Systems](https://mlsanity.com/paper/1812.04186/) (2018)
- [Conditional LSTM-GAN for Melody Generation from Lyrics](https://mlsanity.com/paper/1908.05551/) (2019)
- [miditok: A Python package for MIDI file tokenization](https://mlsanity.com/paper/2310.17202/) (2023)
- [MorpheuS: generating structured music with constrained patterns and tension](https://mlsanity.com/paper/1812.04832/) (2018)
- [MidiCaps: A large-scale MIDI dataset with text captions](https://mlsanity.com/paper/2406.02255/) (2024)
- [Generating Lead Sheets with Affect: A Novel Conditional seq2seq Framework](https://mlsanity.com/paper/2104.13056/) (2021)
- [Exploring the Efficacy of Pre-trained Checkpoints in Text-to-Music Generation Task](https://mlsanity.com/paper/2211.11216/) (2022)
- [Hierarchical Recurrent Neural Networks for Conditional Melody Generation with Long-term Structure](https://mlsanity.com/paper/2102.09794/) (2021)
- [A variational autoencoder for music generation controlled by tonal tension](https://mlsanity.com/paper/2010.06230/) (2020)

## Built on by (most cited first, 19 of 19)

- [JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadata](https://mlsanity.com/paper/2502.07461/) (2025)
- [Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignment](https://mlsanity.com/paper/2505.12669/) (2025)
- [MelodySim: Measuring Melody-aware Music Similarity for Plagiarism Detection](https://mlsanity.com/paper/2505.20979/) (2025)
- [Composer Vector: Style-steering Symbolic Music Generation in a Latent Space](https://mlsanity.com/paper/2604.03333/) (2026)
- [Adaptable Symbolic Music Infilling with MIDI-RWKV](https://mlsanity.com/paper/2506.13001/) (2025)
- [SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton](https://mlsanity.com/paper/2604.25498/) (2026)
- [Not that Groove: Zero-Shot Symbolic Music Editing](https://mlsanity.com/paper/2505.08203/) (2025)
- [On the de-duplication of the Lakh MIDI dataset](https://mlsanity.com/paper/2509.16662/) (2025)
- [MIDI-LLM: Improving Text-to-MIDI Music Generation via Adapting Large Language Models](https://mlsanity.com/paper/2511.03942/) (2025)
- [How Far Can Pretrained LLMs Go in Symbolic Music? Controlled Comparisons of Supervised and Preference-based Adaptation](https://mlsanity.com/paper/2601.22764/) (2026)
- [Efficient Long-Sequence Diffusion Modeling for Symbolic Music Generation](https://mlsanity.com/paper/2603.00576/) (2026)
- [CompLex: Music Theory Lexicon Constructed by Autonomous Agents for Automatic Music Generation](https://mlsanity.com/paper/2508.19603/) (2025)
- [Amadeus: Autoregressive Model with Bidirectional Attribute Modelling for Symbolic Music](https://mlsanity.com/paper/2508.20665/) (2025)
- [HNote: Extending YNote with Hexadecimal Encoding for Fine-Tuning LLMs in Music Modeling](https://mlsanity.com/paper/2509.25694/) (2025)
- [Aligning Generative Music AI with Human Preferences: Methods and Challenges](https://mlsanity.com/paper/2511.15038/) (2025)
- [Story2MIDI: Emotionally Aligned Music Generation from Text](https://mlsanity.com/paper/2512.02192/) (2025)
- [Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping](https://mlsanity.com/paper/2608.03050/) (2026)
- [MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering](https://mlsanity.com/paper/2608.06638/) (2026)
- [Harmonic Representations in Symbolic Music Transformers are Localized and Controllable](https://mlsanity.com/paper/2610.15957/) (2026)

## Abstract (arXiv)

This paper introduces text2midi, an end-to-end model to generate MIDI files from textual descriptions. Leveraging the growing popularity of multimodal generative approaches, text2midi capitalizes on the extensive availability of textual data and the success of large language models (LLMs). Our end-to-end system harnesses the power of LLMs to generate symbolic music in the form of MIDI files. Specifically, we utilize a pretrained LLM encoder to process captions, which then condition an autoregressive transformer decoder to produce MIDI sequences that accurately reflect the provided descriptions. This intuitive and user-friendly method significantly streamlines the music creation process by allowing users to generate music pieces using text prompts. We conduct comprehensive empirical evaluations, incorporating both automated and human studies, that show our model generates MIDI files of high quality that are indeed controllable by text captions that may include music theory terms such as chords, keys, and tempo. We release the code and music samples on our demo page (https://github.com/AMAAI-Lab/Text2midi) for users to interact with text2midi.