mlsanityData to 7 Oct 2026

MuPT: A Generative Symbolic Music Pretrained Transformer

arXiv 2404.06393 ↗First posted 9 Apr 2024cs.SD cs.AI eess.AScited by 32

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore the application of Large Language Models (LLMs) to the pre-training of music. While the prevalent use of MIDI in music modeling is well-established, our findings suggest that LLMs are inherently more compatible with ABC Notation, which aligns more closely with their design and strengths, thereby enhancing the model's performance in musical composition. To address the challenges associated with misaligned measures from different tracks during generation, we propose the development of a Synchronized Multi-Track ABC Notation (SMT-ABC Notation), which aims to preserve coherence across multiple musical tracks. Our contributions include a series of models capable of handling up to 8192 tokens, covering 90% of the symbolic music data in our training set. Furthermore, we explore the implications of the Symbolic Music Scaling Law (SMS Law) on model performance. The results indicate a promising direction for future research in music generation, offering extensive resources for community-led research through our open-source contributions.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

33 of 44 references.

References · 33

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2020GLU Variants Improve Transformercited by 1,724
  8. 2019Root Mean Square Layer Normalizationcited by 1,425
  9. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  10. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  11. 2023Simple and Controllable Music Generationcited by 480
  12. 2023Scaling Data-Constrained Language Modelscited by 391
21 more
  1. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  2. 2022Masked Autoencoders that Listencited by 271
  3. 2021Scaling Laws for Transfercited by 265
  4. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  5. 2021Scaling Laws for Neural Machine Translationcited by 110
  6. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  7. 2024ChatMusician: Understanding and Generating Music Intrinsically with LLMcited by 67
  8. 2023SingSong: Generating musical accompaniments from singingcited by 54
  9. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52
  10. 2023MuseCoco: Generating Symbolic Music from Textcited by 50
  11. 2020Learning Interpretable Representation for Controllable Polyphonic Music Generationcited by 40
  12. 2016Music transcription modelling and composition using deep learningcited by 36
  13. 2019Deep Music Analogy Via Latent Representation Disentanglementcited by 31
  14. 2022Multitrack Music Transformercited by 28
  15. 2023TunesFormer: Forming Irish Tunes with Control Codes by Bar Patchingcited by 20
  16. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 15
  17. 2023CLaMP: Contrastive Language-Music Pre-training for Cross-Modal Symbolic Music Information Retrievalcited by 14
  18. 2022Exploring the Efficacy of Pre-trained Checkpoints in Text-to-Music Generation Taskcited by 13
  19. 2023Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learningcited by 9
  20. 2022MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targetscited by 8
  21. 2023On the Effectiveness of Speech Self-supervised Learning for Musiccited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 12; 2026: 11.

32 papers.

Sort:

By area:

  1. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  2. 2024ComposerX: Multi-Agent Symbolic Music Composition with LLMscited by 40
  3. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  4. 2024Foundation Models for Music: A Surveycited by 34
  5. 2025NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigmscited by 29
  6. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  7. 2025DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matchingcited by 14
  8. 2024CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Modelscited by 12
  9. 2024Creativity in AI: Progresses and Challengescited by 7
  10. 2025Scaling Self-Supervised Representation Learning for Symbolic Piano Performancecited by 7
  11. 2025AutoMV: An Automatic Multi-Agent System for Music Video Generationcited by 7
  12. 2025Moonbeam: A MIDI Foundation Model Using Both Absolute and Relative Music Attributescited by 4
20 more
  1. 2024Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenizationcited by 3
  2. 2025Towards Reliable Large Audio Language Modelcited by 3
  3. 2024Exploring Tokenization Methods for Multitrack Sheet Music Generationcited by 2
  4. 2025Via Score to Performance: Efficient Human-Controllable Long Song Generation with Bar-Level Symbolic Notationcited by 2
  5. 2025PhraseVAE and PhraseLDM: Latent Diffusion for Full-Song Multitrack Symbolic Music Generationcited by 2
  6. 2026BEAT: Tokenizing and Generating Symbolic Music by Uniform Temporal Stepscited by 2
  7. 2025MIDI-LLM: Improving Text-to-MIDI Music Generation via Adapting Large Language Modelscited by 1
  8. 2026How Far Can Pretrained LLMs Go in Symbolic Music? Controlled Comparisons of Supervised and Preference-based Adaptationcited by 1
  9. 2025TOMI: Transforming and Organizing Music Ideas for Multi-Track Compositions with Full-Song Structurecited by 0
  10. 2025Amadeus: Autoregressive Model with Bidirectional Attribute Modelling for Symbolic Musiccited by 0
  11. 2025HNote: Extending YNote with Hexadecimal Encoding for Fine-Tuning LLMs in Music Modelingcited by 0
  12. 2026Whole-Piece Training for Symbolic Music Language Models via Full-Horizon Compressed Recurrencecited by 0
  13. 2026Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fitscited by 0
  14. 2026Exploring LLMs for South Asian Music Understanding and Generationcited by 0
  15. 2026Libretto: Giving LLM Agents a Sense of Musical Structurecited by 0
  16. 2026LEGATO 2: Toward Multimodal Sheet Music Recognition and Understandingcited by 0
  17. 2026How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessnesscited by 0
  18. 2026From local kernels to global form: modeling the emergence of musical contentcited by 0
  19. 2026Which Constraints Are Missing? Ask the Verifier: Graded Rewards for Constraint-Following Music Generationcited by 0
  20. 2026Getting Motif-ated: Controllable AI Compositions from Injected Motif Promptscited by 0