mlsanityData to 7 Oct 2026

Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation

arXiv 2407.21531 ↗First posted 31 Jul 2024cs.SD cs.CL cs.MMcited by 7

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Symbolic Music, akin to language, can be encoded in discrete symbols. Recent research has extended the application of large language models (LLMs) such as GPT-4 and Llama2 to the symbolic music domain including understanding and generation. Yet scant research explores the details of how these LLMs perform on advanced music understanding and conditioned generation, especially from the multi-step reasoning perspective, which is a critical aspect in the conditioned, editable, and interactive human-computer co-creation process. This study conducts a thorough investigation of LLMs' capability and limitations in symbolic music processing. We identify that current LLMs exhibit poor performance in song-level multi-step music reasoning, and typically fail to leverage learned music knowledge when addressing complex musical tasks. An analysis of LLMs' responses highlights distinctly their pros and cons. Our findings suggest achieving advanced musical capability is not intrinsically obtained by LLMs, and future research should focus more on bridging the gap between music knowledge and reasoning, to improve the co-creation experience for musicians.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 27 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

21 of 27 references.

References · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  8. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  9. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  10. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  11. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  12. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
9 more
  1. 2020Investigating Pretrained Language Models for Graph-to-Text Generationcited by 117
  2. 2020A Distributional Approach to Controlled Text Generationcited by 117
  3. 2018Data-to-Text Generation with Content Selection and Planningcited by 106
  4. 2024ChatMusician: Understanding and Generating Music Intrinsically with LLMcited by 67
  5. 2023Natural Language Reasoning, A Surveycited by 61
  6. 2024Creating Suspenseful Stories: Iterative Planning with Large Language Modelscited by 22
  7. 2023MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Modelscited by 20
  8. 2024LLM-based agents for automating the enhancement of user story quality: An early reportcited by 13
  9. 2024ByteComposer: a Human-like Melody Composition Method based on Language Model Agentcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
  2. 2026Large Language Model Reasoning Failurescited by 20
  3. 2024Lines of Thought in Large Language Modelscited by 5
  4. 2025ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Followingcited by 5
  5. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  6. 2025TalkPlayData 2: An Agentic Synthetic Data Pipeline for Multimodal Conversational Music Recommendationcited by 2
  7. 2025Improving BERT for Symbolic Music Understanding Using Token Denoising and Pianoroll Predictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026How Far Can Pretrained LLMs Go in Symbolic Music? Controlled Comparisons of Supervised and Preference-based Adaptation
  2. 2026MIDI-LLaMA: An Instruction-Following Multimodal LLM for Symbolic Music Understanding
  3. 2026Exploring LLMs for South Asian Music Understanding and Generation
  4. 2024Harmonic Reasoning in Large Language Models
  5. 2025TuneGenie: Reasoning-based LLM agents for preferential music generation
  6. 2025The MUSE Benchmark: Probing Music Perception and Auditory Relational Reasoning in Audio LLMS
  7. 2025Evaluating Multimodal Large Language Models on Core Music Perception Tasks
  8. 2025Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning