mlsanityData to 7 Oct 2026

MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response

arXiv 2309.08730 ↗First posted 15 Sep 2023eess.AS cs.AI cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown immense potential in multimodal applications, yet the convergence of textual and musical domains remains not well-explored. To address this gap, we present MusiLingo, a novel system for music caption generation and music-related query responses. MusiLingo employs a single projection layer to align music representations from the pre-trained frozen music audio model MERT with a frozen LLM, bridging the gap between music audio and textual contexts. We train it on an extensive music caption dataset and fine-tune it with instructional data. Due to the scarcity of high-quality music Q&A datasets, we created the MusicInstruct (MI) dataset from captions in the MusicCaps datasets, tailored for open-ended music inquiries. Empirical evaluations demonstrate its competitive performance in generating music captions and composing music-related Q&A pairs. Our introduced dataset enables notable advancements beyond previous ones.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 26 of its 50 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

26 of 50 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  12. 2023Listen, Think, and Understandcited by 206
14 more
  1. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  2. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  3. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
  4. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  5. 2021Audio Captioning Transformercited by 53
  6. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  7. 2020Evaluation of CNN-based Automatic Music Tagging Modelscited by 39
  8. 2022Contrastive Audio-Language Learning for Musiccited by 32
  9. 2021Semi-Supervised Music Tagging Transformercited by 24
  10. 2021MusCaps: Generating Captions for Music Audiocited by 23
  11. 2022Toward Universal Text-to-Music Retrievalcited by 18
  12. 2016Towards Music Captioning: Generating Music Playlist Descriptionscited by 7
  13. 2021Music Playlist Title Generation: A Machine-Translation Approachcited by 5
  14. 2023Music Playlist Title Generation Using Artist Informationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 19; 2026: 13.

45 papers.

Sort:

By area:

  1. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  2. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  3. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 113
  4. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  5. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 66
  6. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 43
  7. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  8. 2024Large Language Models in Cybersecurity: State-of-the-Artcited by 34
  9. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  10. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 15
  11. 2024ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Modelscited by 11
  12. 2024The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Modelscited by 10
33 more
  1. 2024Kun: Answer Polishment for Chinese Self-Alignment with Instruction Back-Translationcited by 9
  2. 2024OpenMU: Your Swiss Army Knife for Music Understandingcited by 9
  3. 2025DIFFA: Large Language Diffusion Models Can Listen and Understandcited by 7
  4. 2026SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processingcited by 7
  5. 2024Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentationcited by 5
  6. 2025Audio Flamingo Sound-CoT Technical Report: Improving Chain-of-Thought Reasoning in Sound Understandingcited by 5
  7. 2025Music Audio-Visual Question Answering Requires Specialized Multimodal Designscited by 4
  8. 2025Towards Reliable Large Audio Language Modelcited by 3
  9. 2025PAL: Probing Audio Encoders via LLMs -- Audio Information Transfer into LLMscited by 3
  10. 2025DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuningcited by 2
  11. 2025Competitive Audio-Language Models with Data-Efficient Single-Stage Training on Public Datacited by 2
  12. 2025Investigating Modality Contribution in Audio LLMs for Musiccited by 2
  13. 2026Rethinking Music Captioning with Music Metadata LLMscited by 2
  14. 2026MusTBench: Benchmarking and Advancing Temporal Grounding in Music LLMscited by 2
  15. 2024Prevailing Research Areas for Music AI in the Era of Foundation Modelscited by 1
  16. 2025Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoderscited by 1
  17. 2025Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluationcited by 1
  18. 2026ALARM: Audio-Language Alignment for Reasoning Modelscited by 1
  19. 2026HumMusQA: A Human-written Music Understanding QA Benchmark Datasetcited by 1
  20. 2026Music Playlist Captioning at Scale with Large Language Modelscited by 1
  21. 2024Benchmarking Sub-Genre Classification For Mainstage Dance Musiccited by 0
  22. 2024Combining Genre Classification and Harmonic-Percussive Features with Diffusion Models for Music-Video Generationcited by 0
  23. 2025Audio-FLAN: An Instruction-Following Dataset for Unified Audio Understanding and Generation of Speech, Music, and Soundcited by 0
  24. 2025TuneGenie: Reasoning-based LLM agents for preferential music generationcited by 0
  25. 2025SonicVerse: Multi-Task Learning for Music Feature-Informed Captioningcited by 0
  26. 2025Advancing the Foundation Model for Music Understandingcited by 0
  27. 2026SongSage: A Large Musical Language Model with Lyric Generative Pre-trainingcited by 0
  28. 2026Art2Mus: Artwork-to-Music Generation via Visual Conditioning and Large-Scale Cross-Modal Alignmentcited by 0
  29. 2026SalsaAgent: A multimodal embodied language model for interactive dance generationcited by 0
  30. 2026Noisy-Channel Minimum Bayes Risk Decodingcited by 0
  31. 2026Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QAcited by 0
  32. 2026Dissonance Spectrum explicitly models perceptual frequency interactions for better music understandingcited by 0
  33. 2026Knowing When Not to Answer: Pseudo-Ensembles for Abstention in Music Audio-Language Modelscited by 0