# Multitrack Music Transformer

arXiv: https://arxiv.org/abs/2207.06983 · HTML: https://mlsanity.com/paper/2207.06983/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 51 references that are in the map.
- First posted on arXiv 14 Jul 2022.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment](https://mlsanity.com/paper/1709.06298/) (2017)
- [Transformer Language Models without Positional Encodings Still Learn Positional Information](https://mlsanity.com/paper/2203.16634/) (2022)
- [C-RNN-GAN: Continuous recurrent neural networks with adversarial training](https://mlsanity.com/paper/1611.09904/) (2016)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [DeepBach: a Steerable Model for Bach Chorales Generation](https://mlsanity.com/paper/1612.01010/) (2016)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation](https://mlsanity.com/paper/1703.10847/) (2017)
- [PopMAG: Pop Music Accompaniment Generation](https://mlsanity.com/paper/2008.07703/) (2020)
- [A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions](https://mlsanity.com/paper/2011.06801/) (2020)
- [LakhNES: Improving multi-instrumental music generation with cross-domain pre-training](https://mlsanity.com/paper/1907.04868/) (2019)
- [The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures](https://mlsanity.com/paper/2008.01307/) (2020)
- [What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding](https://mlsanity.com/paper/2010.04903/) (2020)
- [Deep Learning Techniques for Music Generation -- A Survey](https://mlsanity.com/paper/1709.01620/) (2017)
- [MMM : Exploring Conditional Multi-Track Music Generation with the Transformer](https://mlsanity.com/paper/2008.06048/) (2020)
- [MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE](https://mlsanity.com/paper/2105.04090/) (2021)
- [Theme Transformer: Symbolic Music Generation with Theme-Conditioned Transformer](https://mlsanity.com/paper/2111.04093/) (2021)
- [MusPy: A Toolkit for Symbolic Music Generation](https://mlsanity.com/paper/2008.01951/) (2020)
- [Music transcription modelling and composition using deep learning](https://mlsanity.com/paper/1604.08723/) (2016)
- [Controllable deep melody generation via hierarchical music structure representation](https://mlsanity.com/paper/2109.00663/) (2021)
- [PIANOTREE VAE: Structured Representation Learning for Polyphonic Music](https://mlsanity.com/paper/2008.07118/) (2020)

## Built on by (most cited first, 28 of 28)

- [MuPT: A Generative Symbolic Music Pretrained Transformer](https://mlsanity.com/paper/2404.06393/) (2024)
- [Byte Pair Encoding for Symbolic Music](https://mlsanity.com/paper/2301.11975/) (2023)
- [PDMX: A Large-Scale Public Domain MusicXML Dataset for Symbolic Music Processing](https://mlsanity.com/paper/2409.10831/) (2024)
- [GETMusic: Generating Any Music Tracks with a Unified Representation and Diffusion Framework](https://mlsanity.com/paper/2305.10841/) (2023)
- [A Review of Intelligent Music Generation Systems](https://mlsanity.com/paper/2211.09124/) (2022)
- [Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling](https://mlsanity.com/paper/2310.16334/) (2023)
- [Impact of time and note duration tokenizations on deep learning symbolic music modeling](https://mlsanity.com/paper/2310.08497/) (2023)
- [Nested Music Transformer: Sequentially Decoding Compound Tokens in Symbolic Music and Audio Generation](https://mlsanity.com/paper/2408.01180/) (2024)
- [Moonbeam: A MIDI Foundation Model Using Both Absolute and Relative Music Attributes](https://mlsanity.com/paper/2505.15559/) (2025)
- [QAMRO: Quality-aware Adaptive Margin Ranking Optimization for Human-aligned Assessment of Audio Generation Systems](https://mlsanity.com/paper/2508.08957/) (2025)
- [Multi-view MidiVAE: Fusing Track- and Bar-view Representations for Long Multi-track Symbolic Music Generation](https://mlsanity.com/paper/2401.07532/) (2024)
- [BandCondiNet: Parallel Transformers-based Conditional Popular Music Generation with Multi-View Features](https://mlsanity.com/paper/2407.10462/) (2024)
- [Why Perturbing Symbolic Music is Necessary: Fitting the Distribution of Never-used Notes through a Joint Probabilistic Diffusion Model](https://mlsanity.com/paper/2408.01950/) (2024)
- [Time-Shifted Token Scheduling for Symbolic Music Generation](https://mlsanity.com/paper/2509.23749/) (2025)
- [Towards Real-Time Human-AI Musical Co-Performance: Accompaniment Generation with Latent Diffusion Models and MAX/MSP](https://mlsanity.com/paper/2604.07612/) (2026)
- [SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton](https://mlsanity.com/paper/2604.25498/) (2026)
- [Generating High-quality Symbolic Music Using Fine-grained Discriminators](https://mlsanity.com/paper/2408.01696/) (2024)
- [MMT-BERT: Chord-aware Symbolic Music Generation Based on Multitrack Music Transformer and MusicBERT](https://mlsanity.com/paper/2409.00919/) (2024)
- [AMT-APC: Automatic Piano Cover by Fine-Tuning an Automatic Music Transcription Model](https://mlsanity.com/paper/2409.14086/) (2024)
- [MuseTok: Symbolic Music Tokenization for Generation and Semantic Understanding](https://mlsanity.com/paper/2510.16273/) (2025)
- [ViTex: Visual Texture Control for Multi-Track Symbolic Music Generation via Discrete Diffusion Models](https://mlsanity.com/paper/2603.01984/) (2026)
- [BMdataset: A Musicologically Curated LilyPond Dataset](https://mlsanity.com/paper/2604.10628/) (2026)
- [Latent Space Disentanglement via Activation Steering for Interpretable Attribute Control in Symbolic Music Generation](https://mlsanity.com/paper/2605.31295/) (2026)
- [MusPyExpress: Extending MusPy with Enhanced Expression Text Support](https://mlsanity.com/paper/2608.21678/) (2026)
- [Unsupervised Lead Sheet Generation via Semantic Compression](https://mlsanity.com/paper/2310.10772/) (2023)
- [Do we need more complex representations for structure? A comparison of note duration representation for Music Transformers](https://mlsanity.com/paper/2410.10515/) (2024)
- [Expressive Music Data Processing and Generation](https://mlsanity.com/paper/2503.11896/) (2025)
- [SAGE-Music: Low-Latency Symbolic Music Generation via Attribute-Specialized Key-Value Head Sharing](https://mlsanity.com/paper/2510.00395/) (2025)

## Abstract (arXiv)

Existing approaches for generating multitrack music with transformer models have been limited in terms of the number of instruments, the length of the music segments and slow inference. This is partly due to the memory requirements of the lengthy input sequences necessitated by existing representations. In this work, we propose a new multitrack music representation that allows a diverse set of instruments while keeping a short sequence length. Our proposed Multitrack Music Transformer (MMT) achieves comparable performance with state-of-the-art systems, landing in between two recently proposed models in a subjective listening test, while achieving substantial speedups and memory reductions over both, making the method attractive for real time improvisation or near real time creative applications. Further, we propose a new measure for analyzing musical self-attention and show that the trained model attends more to notes that form a consonant interval with the current note and to notes that are 4N beats away from the current step.