# N-Gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music Understanding

arXiv: https://arxiv.org/abs/2312.08931 · HTML: https://mlsanity.com/paper/2312.08931/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Semantics-aware BERT for Language Understanding](https://mlsanity.com/paper/1909.02209/) (2019)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [POP909: A Pop-song Dataset for Music Arrangement Generation](https://mlsanity.com/paper/2008.07142/) (2020)
- [This Time with Feeling: Learning Expressive Musical Performance](https://mlsanity.com/paper/1808.03715/) (2018)
- [High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times](https://mlsanity.com/paper/2010.01815/) (2020)
- [EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation](https://mlsanity.com/paper/2108.01374/) (2021)
- [GiantMIDI-Piano: A large-scale MIDI dataset for classical piano music](https://mlsanity.com/paper/2010.07061/) (2020)
- [ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations](https://mlsanity.com/paper/1911.00720/) (2019)
- [Computer-Generated Music for Tabletop Role-Playing Games](https://mlsanity.com/paper/2008.07009/) (2020)
- [Text Compression-aided Transformer Encoding](https://mlsanity.com/paper/2102.05951/) (2021)
- [BERT-like Pre-training for Symbolic Piano Music Classification Tasks](https://mlsanity.com/paper/2107.05223/) (2021)
- [Effective Subword Segmentation for Text Comprehension](https://mlsanity.com/paper/1811.02364/) (2018)

## Built on by (most cited first, 5 of 5)

- [NOTA: Multimodal Music Notation Understanding for Visual Large Language Model](https://mlsanity.com/paper/2502.14893/) (2025)
- [Label Drop for Multi-Aspect Relation Modeling in Universal Information Extraction](https://mlsanity.com/paper/2502.12614/) (2025)
- [Let the Model Learn to Feel: Mode-Guided Tonality Injection for Symbolic Music Emotion Recognition](https://mlsanity.com/paper/2512.17946/) (2025)
- [SongSong: A Time Phonograph for Chinese SongCi Music from Thousand of Years Away](https://mlsanity.com/paper/2602.24071/) (2026)
- [N-gram Injection into Transformers for Dynamic Language Model Adaptation in Handwritten Text Recognition](https://mlsanity.com/paper/2603.03930/) (2026)

## Abstract (arXiv)

The first step to apply deep learning techniques for symbolic music understanding is to transform musical pieces (mainly in MIDI format) into sequences of predefined tokens like note pitch, note velocity, and chords. Subsequently, the sequences are fed into a neural sequence model to accomplish specific tasks. Music sequences exhibit strong correlations between adjacent elements, making them prime candidates for N-gram techniques from Natural Language Processing (NLP). Consider classical piano music: specific melodies might recur throughout a piece, with subtle variations each time. In this paper, we propose a novel method, NG-Midiformer, for understanding symbolic music sequences that leverages the N-gram approach. Our method involves first processing music pieces into word-like sequences with our proposed unsupervised compoundation, followed by using our N-gram Transformer encoder, which can effectively incorporate N-gram information to enhance the primary encoder part for better understanding of music sequences. The pre-training process on large-scale music datasets enables the model to thoroughly learn the N-gram information contained within music sequences, and subsequently apply this information for making inferences during the fine-tuning stage. Experiment on various datasets demonstrate the effectiveness of our method and achieved state-of-the-art performance on a series of music understanding downstream tasks. The code and model weights will be released at https://github.com/CinqueOrigin/NG-Midiformer.