mlsanityData to 14 Oct 2026

Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning

arXiv 2308.11276 ↗First posted 22 Aug 2023cs.SD cs.AI cs.CLcited by 77

Not on the map. Why?

Abstract (arXiv)

Text-to-music generation (T2M-Gen) faces a major obstacle due to the scarcity of large-scale publicly available music datasets with natural language captions. To address this, we propose the Music Understanding LLaMA (MU-LLaMA), capable of answering music-related questions and generating captions for music files. Our model utilizes audio representations from a pretrained MERT model to extract music features. However, obtaining a suitable dataset for training the MU-LLaMA model remains challenging, as existing publicly accessible audio question answering datasets lack the necessary depth for open-ended music question answering. To fill this gap, we present a methodology for generating question-answer pairs from existing audio captioning datasets and introduce the MusicQA Dataset designed for answering open-ended music-related questions. The experiments demonstrate that the proposed MU-LLaMA model, trained on our designed MusicQA dataset, achieves outstanding performance in both music question answering and music caption generation across various metrics, outperforming current state-of-the-art (SOTA) models in both fields and offering a promising advancement in the T2M-Gen research field.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 22 Aug 2023.

Builds on

21 of 33 references.

References · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  7. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2020Jukebox: A Generative Model for Musiccited by 542
  10. 2023Simple and Controllable Music Generationcited by 488
  11. 2023MusicLM: Generating Music From Textcited by 448
  12. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
9 more
  1. 2023Listen, Think, and Understandcited by 209
  2. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 181
  3. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
  4. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusioncited by 78
  5. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 64
  6. 2021Audio Captioning Transformercited by 55
  7. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 45
  8. 2021MusCaps: Generating Captions for Music Audiocited by 23
  9. 2022Robust Vector Quantized-Variational Autoencodercited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 22; 2025: 28; 2026: 21.

77 papers.

Sort:

By area:

  1. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  2. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  3. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  4. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  5. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 117
  6. 2024Large Language Models for Time Series: A Surveycited by 114
  7. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  8. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95
  9. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  10. 2023The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluationcited by 68
  11. 2023MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Responsecited by 45
  12. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 44
38 more
  1. 2023Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matchingcited by 42
  2. 2024Foundation Models for Music: A Surveycited by 37
  3. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
  4. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  5. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 31
  6. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 31
  7. 2023LLark: A Multimodal Instruction-Following Language Model for Musiccited by 29
  8. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  9. 2025MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Meshcited by 20
  10. 2024UniMuMo: Unified Text, Music and Motion Generationcited by 19
  11. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 19
  12. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 17
  13. 2024The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Modelscited by 10
  14. 2024OpenMU: Your Swiss Army Knife for Music Understandingcited by 9
  15. 2025DIFFA: Large Language Diffusion Models Can Listen and Understandcited by 8
  16. 2024Audio Dialogues: Dialogues dataset for audio and music understandingcited by 7
  17. 2024Quality-aware Masked Diffusion Transformer for Enhanced Music Generationcited by 6
  18. 2025ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Followingcited by 6
  19. 2024Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentationcited by 5
  20. 2025Audio Flamingo Sound-CoT Technical Report: Improving Chain-of-Thought Reasoning in Sound Understandingcited by 5
  21. 2024A Survey of Foundation Models for Music Understandingcited by 4
  22. 2025NOTA: Multimodal Music Notation Understanding for Visual Large Language Modelcited by 4
  23. 2025A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectivescited by 4
  24. 2025Music Audio-Visual Question Answering Requires Specialized Multimodal Designscited by 4
  25. 2025Revisiting Audio-language Pretraining for Learning General-purpose Audio Representationcited by 4
  26. 2025Towards Reliable Large Audio Language Modelcited by 3
  27. 2025DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuningcited by 2
  28. 2025Competitive Audio-Language Models with Data-Efficient Single-Stage Training on Public Datacited by 2
  29. 2025Investigating Modality Contribution in Audio LLMs for Musiccited by 2
  30. 2026SonicBench: Dissecting the Physical Perception Bottleneck in Large Audio Language Modelscited by 2
  31. 2026Rethinking Music Captioning with Music Metadata LLMscited by 2
  32. 2026MusTBench: Benchmarking and Advancing Temporal Grounding in Music LLMscited by 2
  33. 2024The Interpretation Gap in Text-to-Music Generation Modelscited by 1
  34. 2024Enhancing Audio-Language Models through Self-Supervised Post-Training with Text-Audio Pairscited by 1
  35. 2024Roadmap towards Superhuman Speech Understanding using Large Language Modelscited by 1
  36. 2025Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoderscited by 1
  37. 2025Advancing the Foundation Model for Music Understandingcited by 1
  38. 2025Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Soundcited by 1

50 most cited of 77 All 77 (JSON)