# Anticipatory Music Transformer

arXiv: https://arxiv.org/abs/2306.08620 · HTML: https://mlsanity.com/paper/2306.08620/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 122 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 30 of 75)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)

## Built on by (most cited first, 30 of 36)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models](https://mlsanity.com/paper/2405.09901/) (2024)
- [Aligning Text-to-Music Evaluation with Human Preferences](https://mlsanity.com/paper/2503.16669/) (2025)
- [PDMX: A Large-Scale Public Domain MusicXML Dataset for Symbolic Music Processing](https://mlsanity.com/paper/2409.10831/) (2024)
- [Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modeling](https://mlsanity.com/paper/2504.15071/) (2025)
- [Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling](https://mlsanity.com/paper/2310.16334/) (2023)
- [Scaling Self-Supervised Representation Learning for Symbolic Piano Performance](https://mlsanity.com/paper/2506.23869/) (2025)
- [Composer's Assistant 2: Interactive Multi-Track MIDI Infilling with Fine-Grained User Control](https://mlsanity.com/paper/2407.14700/) (2024)
- [PerTok: Expressive Encoding and Modeling of Symbolic Musical Ideas and Variations](https://mlsanity.com/paper/2410.02060/) (2024)
- [Moonbeam: A MIDI Foundation Model Using Both Absolute and Relative Music Attributes](https://mlsanity.com/paper/2505.15559/) (2025)
- [ImprovNet -- Generating Controllable Musical Improvisations with Iterative Corruption Refinement](https://mlsanity.com/paper/2502.04522/) (2025)
- [BEAT: Tokenizing and Generating Symbolic Music by Uniform Temporal Steps](https://mlsanity.com/paper/2604.19532/) (2026)
- [Computational Copyright: Towards A Royalty Model for Music Generative AI](https://mlsanity.com/paper/2312.06646/) (2023)
- [Local deployment of large-scale music AI models on commodity hardware](https://mlsanity.com/paper/2411.09625/) (2024)
- [PhraseVAE and PhraseLDM: Latent Diffusion for Full-Song Multitrack Symbolic Music Generation](https://mlsanity.com/paper/2512.11348/) (2025)
- [LiveBand: Live Accompaniment Generation in the Audio Domain](https://mlsanity.com/paper/2606.03803/) (2026)
- [Music Proofreading with RefinPaint: Where and How to Modify Compositions given Context](https://mlsanity.com/paper/2407.09099/) (2024)
- [Generating High-quality Symbolic Music Using Fine-grained Discriminators](https://mlsanity.com/paper/2408.01696/) (2024)
- [On the de-duplication of the Lakh MIDI dataset](https://mlsanity.com/paper/2509.16662/) (2025)
- [Supporting Creative Ownership through Deep Learning-Based Music Variation](https://mlsanity.com/paper/2509.25834/) (2025)
- [MuseTok: Symbolic Music Tokenization for Generation and Semantic Understanding](https://mlsanity.com/paper/2510.16273/) (2025)
- [MIDI-LLM: Improving Text-to-MIDI Music Generation via Adapting Large Language Models](https://mlsanity.com/paper/2511.03942/) (2025)
- [ViTex: Visual Texture Control for Multi-Track Symbolic Music Generation via Discrete Diffusion Models](https://mlsanity.com/paper/2603.01984/) (2026)
- [What's a Credit Worth? A Market Framework for Attribution-Aware Compensation in Generative Music](https://mlsanity.com/paper/2607.00641/) (2026)
- [Unsupervised Lead Sheet Generation via Semantic Compression](https://mlsanity.com/paper/2310.10772/) (2023)
- [TOMI: Transforming and Organizing Music Ideas for Multi-Track Compositions with Full-Song Structure](https://mlsanity.com/paper/2506.23094/) (2025)
- [TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization](https://mlsanity.com/paper/2509.00914/) (2025)
- [Structure-Aware Piano Accompaniment via Style Planning and Dataset-Aligned Pattern Retrieval](https://mlsanity.com/paper/2602.15074/) (2026)
- [Libretto: Giving LLM Agents a Sense of Musical Structure](https://mlsanity.com/paper/2606.22708/) (2026)
- [Verifier-Guided Twelve-Tone Composition: A Generate-Verify-Repair Harness for Symbolic Music Generation](https://mlsanity.com/paper/2607.11334/) (2026)

## Abstract (arXiv)

We introduce anticipation: a method for constructing a controllable generative model of a temporal point process (the event process) conditioned asynchronously on realizations of a second, correlated process (the control process). We achieve this by interleaving sequences of events and controls, such that controls appear following stopping times in the event sequence. This work is motivated by problems arising in the control of symbolic music generation. We focus on infilling control tasks, whereby the controls are a subset of the events themselves, and conditional generation completes a sequence of events given the fixed control events. We train anticipatory infilling models using the large and diverse Lakh MIDI music dataset. These models match the performance of autoregressive models for prompted music generation, with the additional capability to perform infilling control tasks, including accompaniment. Human evaluators report that an anticipatory model produces accompaniments with similar musicality to even music composed by humans over a 20-second clip.