# A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling

arXiv: https://arxiv.org/abs/2212.00973 · HTML: https://mlsanity.com/paper/2212.00973/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [This Time with Feeling: Learning Expressive Musical Performance](https://mlsanity.com/paper/1808.03715/) (2018)
- [A Functional Taxonomy of Music Generation Systems](https://mlsanity.com/paper/1812.04186/) (2018)
- [PIANOTREE VAE: Structured Representation Learning for Polyphonic Music](https://mlsanity.com/paper/2008.07118/) (2020)
- [PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Music](https://mlsanity.com/paper/2010.08091/) (2020)
- [Hierarchical Recurrent Neural Networks for Conditional Melody Generation with Long-term Structure](https://mlsanity.com/paper/2102.09794/) (2021)
- [From Context to Concept: Exploring Semantic Relationships in Music with Word2Vec](https://mlsanity.com/paper/1811.12408/) (2018)

## Built on by (most cited first, 17 of 17)

- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [Anticipatory Music Transformer](https://mlsanity.com/paper/2306.08620/) (2023)
- [Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey](https://mlsanity.com/paper/2402.17467/) (2024)
- [Structure-informed Positional Encoding for Music Generation](https://mlsanity.com/paper/2402.13301/) (2024)
- [Moonbeam: A MIDI Foundation Model Using Both Absolute and Relative Music Attributes](https://mlsanity.com/paper/2505.15559/) (2025)
- [SyMuPe: Affective and Controllable Symbolic Music Performance](https://mlsanity.com/paper/2511.03425/) (2025)
- [An Comparative Analysis of Different Pitch and Metrical Grid Encoding Methods in the Task of Sequential Music Generation](https://mlsanity.com/paper/2301.13383/) (2023)
- [Why Perturbing Symbolic Music is Necessary: Fitting the Distribution of Never-used Notes through a Joint Probabilistic Diffusion Model](https://mlsanity.com/paper/2408.01950/) (2024)
- [F-StrIPE: Fast Structure-Informed Positional Encoding for Symbolic Music Generation](https://mlsanity.com/paper/2502.10491/) (2025)
- [Practical and Reproducible Symbolic Music Generation by Large Language Models with Structural Embeddings](https://mlsanity.com/paper/2407.19900/) (2024)
- [Six Dragons Fly Again: Reviving 15th-Century Korean Court Music with Transformers and Novel Encoding](https://mlsanity.com/paper/2408.01096/) (2024)
- [PianoCoRe: Combined and Refined Piano MIDI Dataset](https://mlsanity.com/paper/2605.06627/) (2026)
- [LooPy: A Research-Friendly Mix Framework for Music Information Retrieval on Electronic Dance Music](https://mlsanity.com/paper/2305.01051/) (2023)
- [Of All StrIPEs: Investigating Structure-informed Positional Encoding for Efficient Music Generation](https://mlsanity.com/paper/2504.05364/) (2025)
- [Anchored Cyclic Generation: A Novel Paradigm for Long-Sequence Symbolic Music Generation](https://mlsanity.com/paper/2604.05343/) (2026)
- [Equivariant Music Transformer](https://mlsanity.com/paper/2608.03920/) (2026)
- [How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessness](https://mlsanity.com/paper/2608.18025/) (2026)

## Abstract (arXiv)

Following the success of the transformer architecture in the natural language domain, transformer-like architectures have been widely applied to the domain of symbolic music recently. Symbolic music and text, however, are two different modalities. Symbolic music contains multiple attributes, both absolute attributes (e.g., pitch) and relative attributes (e.g., pitch interval). These relative attributes shape human perception of musical motifs. These important relative attributes, however, are mostly ignored in existing symbolic music modeling methods with the main reason being the lack of a musically-meaningful embedding space where both the absolute and relative embeddings of the symbolic music tokens can be efficiently represented. In this paper, we propose the Fundamental Music Embedding (FME) for symbolic music based on a bias-adjusted sinusoidal encoding within which both the absolute and the relative attributes can be embedded and the fundamental musical properties (e.g., translational invariance) are explicitly preserved. Taking advantage of the proposed FME, we further propose a novel attention mechanism based on the relative index, pitch and onset embeddings (RIPO attention) such that the musical domain knowledge can be fully utilized for symbolic music modeling. Experiment results show that our proposed model: RIPO transformer which utilizes FME and RIPO attention outperforms the state-of-the-art transformers (i.e., music transformer, linear transformer) in a melody completion task. Moreover, using the RIPO transformer in a downstream music generation task, we notice that the notorious degeneration phenomenon no longer exists and the music generated by the RIPO transformer outperforms the music generated by state-of-the-art transformer models in both subjective and objective evaluations.