# Let Network Decide What to Learn: Symbolic Music Understanding Model Based on Large-scale Adversarial Pre-training

arXiv: https://arxiv.org/abs/2407.08306 · HTML: https://mlsanity.com/paper/2407.08306/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 22 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Identifying and Reducing Gender Bias in Word-Level Language Models](https://mlsanity.com/paper/1904.03035/) (2019)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [POP909: A Pop-song Dataset for Music Arrangement Generation](https://mlsanity.com/paper/2008.07142/) (2020)
- [EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation](https://mlsanity.com/paper/2108.01374/) (2021)
- [GiantMIDI-Piano: A large-scale MIDI dataset for classical piano music](https://mlsanity.com/paper/2010.07061/) (2020)
- [PianoBART: Symbolic Piano Music Generation and Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2407.03361/) (2024)
- [To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks](https://mlsanity.com/paper/2006.08671/) (2020)

## Built on by (most cited first, 5 of 5)

- [Automatic Stage Lighting Control: Is it a Rule-Driven Process or Generative Task?](https://mlsanity.com/paper/2506.01482/) (2025)
- [Mode-conditioned music learning and composition: a spiking neural network inspired by neuroscience and psychology](https://mlsanity.com/paper/2411.14773/) (2024)
- [BERT-APC: A Reference-free Framework for Automatic Pitch Correction via Musical Context Inference](https://mlsanity.com/paper/2511.20006/) (2025)
- [AutoFed: Personalized Federated Traffic Prediction via Adaptive Prompt](https://mlsanity.com/paper/2512.24625/) (2025)
- [Stage Light is Sequence2: Multi-Light Control via Imitation Learning](https://mlsanity.com/paper/2605.03660/) (2026)

## Abstract (arXiv)

As a crucial aspect of Music Information Retrieval (MIR), Symbolic Music Understanding (SMU) has garnered significant attention for its potential to assist both musicians and enthusiasts in learning and creating music. Recently, pre-trained language models have been widely adopted in SMU due to the substantial similarities between symbolic music and natural language, as well as the ability of these models to leverage limited music data effectively. However, some studies have shown the common pre-trained methods like Mask Language Model (MLM) may introduce bias issues like racism discrimination in Natural Language Process (NLP) and affects the performance of downstream tasks, which also happens in SMU. This bias often arises when masked tokens cannot be inferred from their context, forcing the model to overfit the training set instead of generalizing. To address this challenge, we propose Adversarial-MidiBERT for SMU, which adaptively determines what to mask during MLM via a masker network, rather than employing random masking. By avoiding the masking of tokens that are difficult to infer from context, our model is better equipped to capture contextual structures and relationships, rather than merely conforming to the training data distribution. We evaluate our method across four SMU tasks, and our approach demonstrates excellent performance in all cases. The code for our model is publicly available at https://github.com/RS2002/Adversarial-MidiBERT .