mlsanityData to 7 Oct 2026

ChatMusician: Understanding and Generating Music Intrinsically with LLM

arXiv 2402.16153 ↗First posted 25 Feb 2024cs.SD cs.AI cs.CLcited by 67

Not on the map. Why?

Abstract (arXiv)

While Large Language Models (LLMs) demonstrate impressive capabilities in text generation, we find that their ability has yet to be generalized to music, humanity's creative language. We introduce ChatMusician, an open-source LLM that integrates intrinsic musical abilities. It is based on continual pre-training and finetuning LLaMA2 on a text-compatible music representation, ABC notation, and the music is treated as a second language. ChatMusician can understand and generate music with a pure text tokenizer without any external multi-modal neural structures or tokenizers. Interestingly, endowing musical abilities does not harm language abilities, even achieving a slightly higher MMLU score. Our model is capable of composing well-structured, full-length music, conditioned on texts, chords, melodies, motifs, musical forms, etc, surpassing GPT-4 baseline. On our meticulously curated college-level music understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and GPT-3.5 on zero-shot setting by a noticeable margin. Our work reveals that LLMs can be an excellent compressor for music, but there remains significant territory to be conquered. We release our 4B token music-language corpora MusicPile, the collected MusicTheoryBench, code, model and demo in GitHub.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 26 of its 47 references that are in the map. First posted on arXiv 25 Feb 2024.

Builds on

26 of 47 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  11. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  12. 2020Jukebox: A Generative Model for Musiccited by 535
14 more
  1. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  2. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  3. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  4. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  5. 2018Music Transformercited by 207
  6. 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 73
  7. 2018This Time with Feeling: Learning Expressive Musical Performancecited by 67
  8. 2023SingSong: Generating musical accompaniments from singingcited by 54
  9. 2023MuseCoco: Generating Symbolic Music from Textcited by 50
  10. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  11. 2022Video Background Music Generation: Dataset, Method and Evaluationcited by 28
  12. 2020Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositionscited by 16
  13. 2022Exploring the Efficacy of Pre-trained Checkpoints in Text-to-Music Generation Taskcited by 13
  14. 2022What is missing in deep music generation? A study of repetition and structure in popular musiccited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 28; 2026: 17.

67 papers.

Sort:

By area:

  1. 2025Agentic Large Language Models, a surveycited by 105
  2. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  3. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 43
  4. 2024ComposerX: Multi-Agent Symbolic Music Composition with LLMscited by 40
  5. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 39
  6. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  7. 2024Foundation Models for Music: A Surveycited by 34
  8. 2024MuPT: A Generative Symbolic Music Pretrained Transformercited by 32
  9. 2024COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representationscited by 14
  10. 2024MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptionscited by 13
  11. 2024Do Music Generation Models Encode Music Theory?cited by 12
  12. 2024Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Surveycited by 10
38 more
  1. 2024The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Modelscited by 10
  2. 2024Fourier Head: Helping Large Language Models Learn Complex Probability Distributionscited by 10
  3. 2025Survey on the Evaluation of Generative Models in Musiccited by 10
  4. 2025WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoningcited by 10
  5. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 10
  6. 2024SongComposer: A Large Language Model for Lyric and Melody Generation in Song Compositioncited by 9
  7. 2024Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problemscited by 9
  8. 2024Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generationcited by 7
  9. 2024Creativity in AI: Progresses and Challengescited by 7
  10. 2025SingMOS-Pro: An Comprehensive Benchmark for Singing Quality Assessmentcited by 7
  11. 2025On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Accelerationcited by 6
  12. 2025Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoningcited by 6
  13. 2026AuTAgent: A Reinforcement Learning Framework for Tool-Augmented Audio Reasoningcited by 6
  14. 2025ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Followingcited by 5
  15. 2024A Survey of Foundation Models for Music Understandingcited by 4
  16. 2025NOTA: Multimodal Music Notation Understanding for Visual Large Language Modelcited by 4
  17. 2025KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoningcited by 4
  18. 2025Moonbeam: A MIDI Foundation Model Using Both Absolute and Relative Music Attributescited by 4
  19. 2025Music Audio-Visual Question Answering Requires Specialized Multimodal Designscited by 4
  20. 2025MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Modelscited by 4
  21. 2025Towards Reliable Large Audio Language Modelcited by 3
  22. 2025Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisonscited by 2
  23. 2025Via Score to Performance: Efficient Human-Controllable Long Song Generation with Bar-Level Symbolic Notationcited by 2
  24. 2025CSyMR: Benchmarking Compositional Music Information Retrieval in Symbolic Music Reasoningcited by 2
  25. 2026Rethinking Music Captioning with Music Metadata LLMscited by 2
  26. 2026Composer Vector: Style-steering Symbolic Music Generation in a Latent Spacecited by 2
  27. 2026BEAT: Tokenizing and Generating Symbolic Music by Uniform Temporal Stepscited by 2
  28. 2024The Interpretation Gap in Text-to-Music Generation Modelscited by 1
  29. 2024Emotion-Guided Image to Music Generationcited by 1
  30. 2025TRIZ Agents: A Multi-Agent LLM Approach for TRIZ-Based Innovationcited by 1
  31. 2025Content filtering methods for music recommendation: A reviewcited by 1
  32. 2025CoComposer: LLM Multi-agent Collaborative Music Compositioncited by 1
  33. 2025MIDI-LLM: Improving Text-to-MIDI Music Generation via Adapting Large Language Modelscited by 1
  34. 2025Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Soundcited by 1
  35. 2025Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluationcited by 1
  36. 2026How Far Can Pretrained LLMs Go in Symbolic Music? Controlled Comparisons of Supervised and Preference-based Adaptationcited by 1
  37. 2026HumMusQA: A Human-written Music Understanding QA Benchmark Datasetcited by 1
  38. 2026AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generationcited by 1

50 most cited of 67 All 67 (JSON)