mlsanityData to 14 Oct 2026

MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training

arXiv 2106.05630 ↗First posted 10 Jun 2021cs.SD cs.CL cs.IRcited by 89

Not on the map. Why?

Abstract (arXiv)

Symbolic music understanding, which refers to the understanding of music from the symbolic data (e.g., MIDI format, but not audio), covers many music applications such as genre classification, emotion classification, and music pieces matching. While good music representations are beneficial for these applications, the lack of training data hinders representation learning. Inspired by the success of pre-training models in natural language processing, in this paper, we develop MusicBERT, a large-scale pre-trained model for music understanding. To this end, we construct a large-scale symbolic music corpus that contains more than 1 million music songs. Since symbolic music contains more structural (e.g., bar, position) and diverse information (e.g., tempo, instrument, and pitch), simply adopting the pre-training techniques from NLP to symbolic music only brings marginal gains. Therefore, we design several mechanisms, including OctupleMIDI encoding and bar-level masking strategy, to enhance pre-training with symbolic music data. Experiments demonstrate the advantages of MusicBERT on four music understanding tasks, including melody completion, accompaniment suggestion, genre classification, and style classification. Ablation studies also verify the effectiveness of our designs of OctupleMIDI encoding and bar-level masking strategy in MusicBERT.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

26 of 37 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  9. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  10. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  11. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
  12. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 217
14 more
  1. 2018Music Transformercited by 211
  2. 2021Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphscited by 110
  3. 2018This Time with Feeling: Learning Expressive Musical Performancecited by 68
  4. 2020PopMAG: Pop Music Accompaniment Generationcited by 68
  5. 2020High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Timescited by 61
  6. 2020A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directionscited by 56
  7. 2020SongMASS: Automatic Song Writing with Pre-training and Alignment Constraintcited by 41
  8. 2020GiantMIDI-Piano: A large-scale MIDI dataset for classical piano musiccited by 37
  9. 2018Symbolic Music Genre Transfer with CycleGANcited by 28
  10. 2020PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Musiccited by 24
  11. 2018From Context to Concept: Exploring Semantic Relationships in Music with Word2Veccited by 9
  12. 2017Multi-label Music Genre Classification from Audio, Text, and Images Using Deep Featurescited by 7
  13. 2019On large-scale genre classification in symbolically encoded music by automatic identification of repeating patternscited by 6
  14. 2017Modeling Musical Context with Word2veccited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 10; 2023: 18; 2024: 19; 2025: 26; 2026: 15.

89 papers.

Sort:

By area:

  1. 2023MuseCoco: Generating Symbolic Music from Textcited by 52
  2. 2022Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generationcited by 49
  3. 2023Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer modelcited by 37
  4. 2023Anticipatory Music Transformercited by 36
  5. 2022Symphony Generation with Permutation Invariant Language Modelcited by 35
  6. 2022Multitrack Music Transformercited by 28
  7. 2023miditok: A Python package for MIDI file tokenizationcited by 27
  8. 2024Beyond Language Models: Byte Models are Digital World Simulatorscited by 22
  9. 2023MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Modelscited by 20
  10. 2023Byte Pair Encoding for Symbolic Musiccited by 19
  11. 2022MusIAC: An extensible generative framework for Music Infilling Applications with multi-level Controlcited by 17
  12. 2023Discrete Diffusion Probabilistic Models for Symbolic Music Generationcited by 16
38 more
  1. 2024PDMX: A Large-Scale Public Domain MusicXML Dataset for Symbolic Music Processingcited by 15
  2. 2023GETMusic: Generating Any Music Tracks with a Unified Representation and Diffusion Frameworkcited by 13
  3. 2021BERT-like Pre-training for Symbolic Piano Music Classification Taskscited by 12
  4. 2024PianoBART: Symbolic Piano Music Generation and Understanding with Large-Scale Pre-Trainingcited by 12
  5. 2024CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Modelscited by 12
  6. 2022Contrastive Learning with Positive-Negative Frame Mask for Music Representationcited by 10
  7. 2023MelodyGLM: Multi-task Pre-training for Symbolic Melody Generationcited by 10
  8. 2024Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Surveycited by 10
  9. 2024The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Modelscited by 10
  10. 2025Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modelingcited by 10
  11. 2023Can Self-Supervised Neural Representations Pre-Trained on Human Speech distinguish Animal Callers?cited by 9
  12. 2024End-to-end Piano Performance-MIDI to Score Conversion with Transformerscited by 9
  13. 2025Scaling Self-Supervised Representation Learning for Symbolic Piano Performancecited by 9
  14. 2025MIDI-VALLE: Improving Expressive Piano Performance Synthesis Through Neural Codec Language Modellingcited by 8
  15. 2022YM2413-MDB: A Multi-Instrumental FM Video Game Music Dataset with Emotion Annotationscited by 7
  16. 2023Reconstructing Human Expressiveness in Piano Performances with a Transformer Networkcited by 7
  17. 2025The GigaMIDI Dataset with Features for Expressive Music Performance Detectioncited by 7
  18. 2022Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generationcited by 6
  19. 2023Impact of time and note duration tokenizations on deep learning symbolic music modelingcited by 6
  20. 2025ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Followingcited by 6
  21. 2023WuYun: Exploring hierarchical skeleton-guided melody generation using knowledge-enhanced deep learningcited by 5
  22. 2023N-Gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music Understandingcited by 5
  23. 2024Let Network Decide What to Learn: Symbolic Music Understanding Model Based on Large-scale Adversarial Pre-trainingcited by 5
  24. 2024Nested Music Transformer: Sequentially Decoding Compound Tokens in Symbolic Music and Audio Generationcited by 5
  25. 2024The NES Video-Music Database: A Dataset of Symbolic Video Game Music Paired with Gameplay Videoscited by 4
  26. 2024A Survey of Foundation Models for Music Understandingcited by 4
  27. 2025A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectivescited by 4
  28. 2025Moonbeam: A MIDI Foundation Model Using Both Absolute and Relative Music Attributescited by 4
  29. 2025Automatic Stage Lighting Control: Is it a Rule-Driven Process or Generative Task?cited by 4
  30. 2022A Novel Multi-Task Learning Method for Symbolic Music Emotion Recognitioncited by 3
  31. 2022A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectivescited by 3
  32. 2023Language Models are Drummers: Drum Composition with Natural Language Pre-Trainingcited by 3
  33. 2024Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenizationcited by 3
  34. 2025Exploring the Needs of Practising Musicians in Co-Creative AI Through Co-Designcited by 3
  35. 2025SyMuPe: Affective and Controllable Symbolic Music Performancecited by 3
  36. 2023An Comparative Analysis of Different Pitch and Metrical Grid Encoding Methods in the Task of Sequential Music Generationcited by 2
  37. 2024Multi-view MidiVAE: Fusing Track- and Bar-view Representations for Long Multi-track Symbolic Music Generationcited by 2
  38. 2024MuDiT & MuSiT: Alignment with Colloquial Expression in Description-to-Song Generationcited by 2

50 most cited of 89 All 89 (JSON)