mlsanityData to 7 Oct 2026

A Survey of Foundation Models for Music Understanding

arXiv 2409.09601 ↗First posted 15 Sep 2024cs.SD cs.AI cs.MMcited by 4

Not on the map. Why?

Abstract (arXiv)

Music is essential in daily life, fulfilling emotional and entertainment needs, and connecting us personally, socially, and culturally. A better understanding of music can enhance our emotions, cognitive skills, and cultural connections. The rapid advancement of artificial intelligence (AI) has introduced new ways to analyze music, aiming to replicate human understanding of music and provide related services. While the traditional models focused on audio features and simple tasks, the recent development of large language models (LLMs) and foundation models (FMs), which excel in various fields by integrating semantic information and demonstrating strong reasoning abilities, could capture complex musical features and patterns, integrate music with language and incorporate rich musical, emotional and psychological knowledge. Therefore, they have the potential in handling complex music understanding tasks from a semantic perspective, producing outputs closer to human perception. This work, to our best knowledge, is one of the early reviews of the intersection of AI techniques and music understanding. We investigated, analyzed, and tested recent large-scale music foundation models in respect of their music comprehension abilities. We also discussed their limitations and proposed possible future directions, offering insights for researchers in this field.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 49 of its 102 references that are in the map. First posted on arXiv 15 Sep 2024.

Builds on

49 of 102 references.

References · 49

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2022Generative Adversarial Networkscited by 5,603
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  12. 2021ViViT: A Video Vision Transformercited by 1,610
37 more
  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  3. 2023A Survey on Multimodal Large Language Modelscited by 898
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  6. 2021AST: Audio Spectrogram Transformercited by 634
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  10. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  11. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  12. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  13. 2023MusicLM: Generating Music From Textcited by 442
  14. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  15. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  16. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  17. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  18. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  19. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  20. 2023Listen, Think, and Understandcited by 206
  21. 2016FMA: A Dataset For Music Analysiscited by 205
  22. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  23. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  24. 2021MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Trainingcited by 89
  25. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  26. 2024ChatMusician: Understanding and Generating Music Intrinsically with LLMcited by 67
  27. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  28. 2021Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processingcited by 57
  29. 2023Psy-LLM: Scaling up Global Mental Health Psychological Services with AI-based Large Language Modelscited by 36
  30. 2021Rethinking Supervised Pre-training for Better Downstream Transferringcited by 35
  31. 2024Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuningcited by 31
  32. 2021MusCaps: Generating Captions for Music Audiocited by 23
  33. 2024ModaVerse: Efficiently Transforming Modalities with LLMscited by 18
  34. 2023Adapting a ConvNeXt model to audio classification on AudioSetcited by 13
  35. 2022A Review of Intelligent Music Generation Systemscited by 11
  36. 2023Self-Specialization: Uncovering Latent Expertise within Large Language Modelscited by 5
  37. 2022Challenges of Artificial Intelligence -- From Machine Learning and Computer Vision to Emotional Intelligencecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  2. 2026Large Language Model Reasoning Failurescited by 20
  3. 2025CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learningcited by 1
  4. 2024Music Genre Classification using Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Foundation Models for Music: A Survey
  2. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models
  3. 2024Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation
  4. 2025The MUSE Benchmark: Probing Music Perception and Auditory Relational Reasoning in Audio LLMS
  5. 2025Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores
  6. 2025Music Flamingo: Scaling Music Understanding in Audio Language Models
  7. 2026MIDI-LLaMA: An Instruction-Following Multimodal LLM for Symbolic Music Understanding
  8. 2025Artificial Intelligence Agents in Music Analysis: An Integrative Perspective Based on Two Use Cases