# Music2Video: Automatic Generation of Music Video with fusion of audio and text

arXiv: https://arxiv.org/abs/2201.03809 · HTML: https://mlsanity.com/paper/2201.03809/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2022.

## Built on by (most cited first, 3 of 3)

- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Audio-guided Album Cover Art Generation with Genetic Algorithms](https://mlsanity.com/paper/2207.07162/) (2022)
- [AgentMV: A State-Guided Multi-Agent Framework for Budget-Aware Music Video Generation](https://mlsanity.com/paper/2605.10723/) (2026)

## Abstract (arXiv)

Creation of images using generative adversarial networks has been widely adapted into multi-modal regime with the advent of multi-modal representation models pre-trained on large corpus. Various modalities sharing a common representation space could be utilized to guide the generative models to create images from text or even from audio source. Departing from the previous methods that solely rely on either text or audio, we exploit the expressiveness of both modality. Based on the fusion of text and audio, we create video whose content is consistent with the distinct modalities that are provided. A simple approach to automatically segment the video into variable length intervals and maintain time consistency in generated video is part of our method. Our proposed framework for generating music video shows promising results in application level where users can interactively feed in music source and text source to create artistic music videos. Our code is available at https://github.com/joeljang/music2video.