# Structure-informed Positional Encoding for Music Generation

arXiv: https://arxiv.org/abs/2402.13301 · HTML: https://mlsanity.com/paper/2402.13301/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel](https://mlsanity.com/paper/1908.11775/) (2019)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [Transformer Language Models without Positional Encodings Still Learn Positional Information](https://mlsanity.com/paper/2203.16634/) (2022)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [POP909: A Pop-song Dataset for Music Arrangement Generation](https://mlsanity.com/paper/2008.07142/) (2020)
- [PopMAG: Pop Music Accompaniment Generation](https://mlsanity.com/paper/2008.07703/) (2020)
- [The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures](https://mlsanity.com/paper/2008.01307/) (2020)
- [Relative Positional Encoding for Transformers with Linear Complexity](https://mlsanity.com/paper/2105.08399/) (2021)
- [Symphony Generation with Permutation Invariant Language Model](https://mlsanity.com/paper/2205.05448/) (2022)
- [Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training](https://mlsanity.com/paper/2109.07306/) (2021)
- [Automatic Analysis and Influence of Hierarchical Structure on Melody, Rhythm and Harmony in Popular Music](https://mlsanity.com/paper/2010.07518/) (2020)
- [A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling](https://mlsanity.com/paper/2212.00973/) (2022)
- [Which transformer architecture fits my data? A vocabulary bottleneck in self-attention](https://mlsanity.com/paper/2105.03928/) (2021)
- [How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection](https://mlsanity.com/paper/1902.10339/) (2019)
- [Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers](https://mlsanity.com/paper/2302.01925/) (2023)

## Built on by (most cited first, 5 of 5)

- [F-StrIPE: Fast Structure-Informed Positional Encoding for Symbolic Music Generation](https://mlsanity.com/paper/2502.10491/) (2025)
- [Diffusion-based Symbolic Music Generation with Structured State Space Models](https://mlsanity.com/paper/2507.20128/) (2025)
- [Efficient Long-Sequence Diffusion Modeling for Symbolic Music Generation](https://mlsanity.com/paper/2603.00576/) (2026)
- [Of All StrIPEs: Investigating Structure-informed Positional Encoding for Efficient Music Generation](https://mlsanity.com/paper/2504.05364/) (2025)
- [SAGE-Music: Low-Latency Symbolic Music Generation via Attribute-Specialized Key-Value Head Sharing](https://mlsanity.com/paper/2510.00395/) (2025)

## Abstract (arXiv)

Music generated by deep learning methods often suffers from a lack of coherence and long-term organization. Yet, multi-scale hierarchical structure is a distinctive feature of music signals. To leverage this information, we propose a structure-informed positional encoding framework for music generation with Transformers. We design three variants in terms of absolute, relative and non-stationary positional information. We comprehensively test them on two symbolic music generation tasks: next-timestep prediction and accompaniment generation. As a comparison, we choose multiple baselines from the literature and demonstrate the merits of our methods using several musically-motivated evaluation metrics. In particular, our methods improve the melodic and structural consistency of the generated pieces.