# PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Music

arXiv: https://arxiv.org/abs/2010.08091 · HTML: https://mlsanity.com/paper/2010.08091/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2020.

## Built on by (most cited first, 24 of 24)

- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [Neuro-Symbolic AI in 2024: A Systematic Review](https://mlsanity.com/paper/2501.05435/) (2025)
- [Symphony Generation with Permutation Invariant Language Model](https://mlsanity.com/paper/2205.05448/) (2022)
- [TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Method](https://mlsanity.com/paper/2109.09617/) (2021)
- [A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling](https://mlsanity.com/paper/2212.00973/) (2022)
- [Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey](https://mlsanity.com/paper/2402.17467/) (2024)
- [The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2406.15885/) (2024)
- [YM2413-MDB: A Multi-Instrumental FM Video Game Music Dataset with Emotion Annotations](https://mlsanity.com/paper/2211.07131/) (2022)
- [Addressing the Vulnerability of NMT in Input Perturbations](https://mlsanity.com/paper/2104.09810/) (2021)
- [Towards Standard Criteria for human evaluation of Chatbots: A Survey](https://mlsanity.com/paper/2105.11197/) (2021)
- [Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation](https://mlsanity.com/paper/2208.05697/) (2022)
- [MRC-LSTM: A Hybrid Approach of Multi-scale Residual CNN and LSTM to Predict Bitcoin Price](https://mlsanity.com/paper/2105.00707/) (2021)
- [Pitchclass2vec: Symbolic Music Structure Segmentation with Chord Embeddings](https://mlsanity.com/paper/2303.15306/) (2023)
- [Language Models are Drummers: Drum Composition with Natural Language Pre-Training](https://mlsanity.com/paper/2301.01162/) (2023)
- [An Comparative Analysis of Different Pitch and Metrical Grid Encoding Methods in the Task of Sequential Music Generation](https://mlsanity.com/paper/2301.13383/) (2023)
- [BandCondiNet: Parallel Transformers-based Conditional Popular Music Generation with Multi-View Features](https://mlsanity.com/paper/2407.10462/) (2024)
- [Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores](https://mlsanity.com/paper/2511.20697/) (2025)
- [MagicPai at SemEval-2021 Task 7: Method for Detecting and Rating Humor Based on Multi-Task Adversarial Training](https://mlsanity.com/paper/2104.10336/) (2021)
- [On the de-duplication of the Lakh MIDI dataset](https://mlsanity.com/paper/2509.16662/) (2025)
- [Prediction, Selection, and Generation: Exploration of Knowledge-Driven Conversation System](https://mlsanity.com/paper/2104.11454/) (2021)
- [RefBERT: Compressing BERT by Referencing to Pre-computed Representations](https://mlsanity.com/paper/2106.08898/) (2021)
- [Learning Music-Dance Representations through Explicit-Implicit Rhythm Synchronization](https://mlsanity.com/paper/2207.03190/) (2022)
- [Large Language Models' Internal Perception of Symbolic Music](https://mlsanity.com/paper/2507.12808/) (2025)
- [Balancing Privacy and Efficiency: Music Information Retrieval via Additive Homomorphic Encryption](https://mlsanity.com/paper/2508.07044/) (2025)

## Abstract (arXiv)

Definitive embeddings remain a fundamental challenge of computational musicology for symbolic music in deep learning today. Analogous to natural language, music can be modeled as a sequence of tokens. This motivates the majority of existing solutions to explore the utilization of word embedding models to build music embeddings. However, music differs from natural languages in two key aspects: (1) musical token is multi-faceted -- it comprises of pitch, rhythm and dynamics information; and (2) musical context is two-dimensional -- each musical token is dependent on both melodic and harmonic contexts. In this work, we provide a comprehensive solution by proposing a novel framework named PiRhDy that integrates pitch, rhythm, and dynamics information seamlessly. PiRhDy adopts a hierarchical strategy which can be decomposed into two steps: (1) token (i.e., note event) modeling, which separately represents pitch, rhythm, and dynamics and integrates them into a single token embedding; and (2) context modeling, which utilizes melodic and harmonic knowledge to train the token embedding. A thorough study was made on each component and sub-strategy of PiRhDy. We further validate our embeddings in three downstream tasks -- melody completion, accompaniment suggestion, and genre classification. Results indicate a significant advancement of the neural approach towards symbolic music as well as PiRhDy's potential as a pretrained tool for a broad range of symbolic music applications.