# Theme Transformer: Symbolic Music Generation with Theme-Conditioned Transformer

arXiv: https://arxiv.org/abs/2111.04093 · HTML: https://mlsanity.com/paper/2111.04093/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2021.

## Built on by (most cited first, 30 of 40)

- [Anticipatory Music Transformer](https://mlsanity.com/paper/2306.08620/) (2023)
- [Multitrack Music Transformer](https://mlsanity.com/paper/2207.06983/) (2022)
- [Compose & Embellish: Well-Structured Piano Performance Generation via A Two-Stage Approach](https://mlsanity.com/paper/2209.08212/) (2022)
- [Frechet Music Distance: A Metric For Generative Symbolic Music Evaluation](https://mlsanity.com/paper/2412.07948/) (2024)
- [GTR-CTRL: Instrument and Genre Conditioning for Guitar-Focused Music Generation with Transformers](https://mlsanity.com/paper/2302.05393/) (2023)
- [MelodyGLM: Multi-task Pre-training for Symbolic Melody Generation](https://mlsanity.com/paper/2309.10738/) (2023)
- [Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey](https://mlsanity.com/paper/2402.17467/) (2024)
- [A Large-Scale Simulation on Large Language Models for Decision-Making in Political Science](https://mlsanity.com/paper/2412.15291/) (2024)
- [ComMU: Dataset for Combinatorial Music Generation](https://mlsanity.com/paper/2211.09385/) (2022)
- [Motifs, Phrases, and Beyond: The Modelling of Structure in Symbolic Music Generation](https://mlsanity.com/paper/2403.07995/) (2024)
- [The GigaMIDI Dataset with Features for Expressive Music Performance Detection](https://mlsanity.com/paper/2502.17726/) (2025)
- [A Survey of Generative AI for Intelligent Transportation Systems: Road Transportation Perspective](https://mlsanity.com/paper/2312.08248/) (2023)
- [An Order-Complexity Aesthetic Assessment Model for Aesthetic-aware Music Recommendation](https://mlsanity.com/paper/2402.08300/) (2024)
- [MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks](https://mlsanity.com/paper/2208.14345/) (2022)
- [EmoGen: Eliminating Subjective Bias in Emotional Music Generation](https://mlsanity.com/paper/2307.01229/) (2023)
- [Melody Infilling with User-Provided Structural Context](https://mlsanity.com/paper/2210.02829/) (2022)
- [PerTok: Expressive Encoding and Modeling of Symbolic Musical Ideas and Variations](https://mlsanity.com/paper/2410.02060/) (2024)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)
- [A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives](https://mlsanity.com/paper/2210.13944/) (2022)
- [Diffusion-based Symbolic Music Generation with Structured State Space Models](https://mlsanity.com/paper/2507.20128/) (2025)
- [Time-Shifted Token Scheduling for Symbolic Music Generation](https://mlsanity.com/paper/2509.23749/) (2025)
- [Segment-Factorized Full-Song Generation on Symbolic Piano Music](https://mlsanity.com/paper/2510.05881/) (2025)
- [SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton](https://mlsanity.com/paper/2604.25498/) (2026)
- [An Order-Complexity Model for Aesthetic Quality Assessment of Symbolic Homophony Music Scores](https://mlsanity.com/paper/2301.05908/) (2023)
- [Motif-Centric Representation Learning for Symbolic Music](https://mlsanity.com/paper/2309.10597/) (2023)
- [Music Proofreading with RefinPaint: Where and How to Modify Compositions given Context](https://mlsanity.com/paper/2407.09099/) (2024)
- [Practical and Reproducible Symbolic Music Generation by Large Language Models with Structural Embeddings](https://mlsanity.com/paper/2407.19900/) (2024)
- [Six Dragons Fly Again: Reviving 15th-Century Korean Court Music with Transformers and Novel Encoding](https://mlsanity.com/paper/2408.01096/) (2024)
- [Generating High-quality Symbolic Music Using Fine-grained Discriminators](https://mlsanity.com/paper/2408.01696/) (2024)
- [Genre Controlled Music Generation via Activation Steering](https://mlsanity.com/paper/2506.10225/) (2025)

## Abstract (arXiv)

Attention-based Transformer models have been increasingly employed for automatic music generation. To condition the generation process of such a model with a user-specified sequence, a popular approach is to take that conditioning sequence as a priming sequence and ask a Transformer decoder to generate a continuation. However, this prompt-based conditioning cannot guarantee that the conditioning sequence would develop or even simply repeat itself in the generated continuation. In this paper, we propose an alternative conditioning approach, called theme-based conditioning, that explicitly trains the Transformer to treat the conditioning sequence as a thematic material that has to manifest itself multiple times in its generation result. This is achieved with two main technical contributions. First, we propose a deep learning-based approach that uses contrastive representation learning and clustering to automatically retrieve thematic materials from music pieces in the training data. Second, we propose a novel gated parallel attention module to be used in a sequence-to-sequence (seq2seq) encoder/decoder architecture to more effectively account for a given conditioning thematic material in the generation process of the Transformer decoder. We report on objective and subjective evaluations of variants of the proposed Theme Transformer and the conventional prompt-based baseline, showing that our best model can generate, to some extent, polyphonic pop piano music with repetition and plausible variations of a given condition.