# M2UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models

arXiv: https://arxiv.org/abs/2311.11255 · HTML: https://mlsanity.com/paper/2311.11255/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2023.

## Built on by (most cited first, 24 of 24)

- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [The Revolution of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2402.12451/) (2024)
- [MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models](https://mlsanity.com/paper/2408.01337/) (2024)
- [VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling](https://mlsanity.com/paper/2406.04321/) (2024)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models](https://mlsanity.com/paper/2402.06178/) (2024)
- [Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey](https://mlsanity.com/paper/2412.18619/) (2024)
- [BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5](https://mlsanity.com/paper/2406.19954/) (2024)
- [MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization](https://mlsanity.com/paper/2410.12957/) (2024)
- [Audio Prompt Adapter: Unleashing Music Editing Abilities for Text-to-Music with Lightweight Finetuning](https://mlsanity.com/paper/2407.16564/) (2024)
- [OpenMU: Your Swiss Army Knife for Music Understanding](https://mlsanity.com/paper/2410.15573/) (2024)
- [Mozart's Touch: A Lightweight Multi-modal Music Generation Framework Based on Pre-Trained Large Models](https://mlsanity.com/paper/2405.02801/) (2024)
- [FilmComposer: LLM-Driven Music Production for Silent Film Clips](https://mlsanity.com/paper/2503.08147/) (2025)
- [Vision-to-Music Generation: A Survey](https://mlsanity.com/paper/2503.21254/) (2025)
- [PAL: Probing Audio Encoders via LLMs -- Audio Information Transfer into LLMs](https://mlsanity.com/paper/2506.10423/) (2025)
- [Rethinking Music Captioning with Music Metadata LLMs](https://mlsanity.com/paper/2602.03023/) (2026)
- [V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generation](https://mlsanity.com/paper/2603.11042/) (2026)
- [Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections](https://mlsanity.com/paper/2608.11576/) (2026)
- [BNMusic: Blending Environmental Noises into Personalized Music](https://mlsanity.com/paper/2506.10754/) (2025)
- [MusRec: Zero-Shot Text-to-Music Editing via Rectified Flow and Diffusion Transformers](https://mlsanity.com/paper/2511.04376/) (2025)
- [Art2Music: Generating Music for Art Images with Multi-modal Feeling Alignment](https://mlsanity.com/paper/2512.00120/) (2025)
- [FlowSonic: Stable Zero-Shot Music Editing via High-Order Trajectory Integration](https://mlsanity.com/paper/2607.17526/) (2026)
- [Mizar: A 159M-Parameter Audio-Language Model for Audio Understanding](https://mlsanity.com/paper/2609.28344/) (2026)

## Abstract (arXiv)

The current landscape of research leveraging large language models (LLMs) is experiencing a surge. Many works harness the powerful reasoning capabilities of these models to comprehend various modalities, such as text, speech, images, videos, etc. They also utilize LLMs to understand human intention and generate desired outputs like images, videos, and music. However, research that combines both understanding and generation using LLMs is still limited and in its nascent stage. To address this gap, we introduce a Multi-modal Music Understanding and Generation (M2UGen) framework that integrates LLM's abilities to comprehend and generate music for different modalities. The M2UGen framework is purpose-built to unlock creative potential from diverse sources of inspiration, encompassing music, image, and video through the use of pretrained MERT, ViT, and ViViT models, respectively. To enable music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging multi-modal understanding and music generation is accomplished through the integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA model to generate extensive datasets that support text/image/video-to-music generation, facilitating the training of our M2UGen framework. We conduct a thorough evaluation of our proposed framework. The experimental results demonstrate that our model achieves or surpasses the performance of the current state-of-the-art models.