# From Sound to Sight: Towards AI-authored Music Videos

arXiv: https://arxiv.org/abs/2509.00029 · HTML: https://mlsanity.com/paper/2509.00029/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 75 references that are in the map.
- First posted on arXiv 20 Aug 2025.

## Builds on (most cited first, 22 of 22)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities](https://mlsanity.com/paper/2406.11768/) (2024)
- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://mlsanity.com/paper/2212.07981/) (2022)
- [Are Emergent Abilities in Large Language Models just In-Context Learning?](https://mlsanity.com/paper/2309.01809/) (2023)
- [CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Norms](https://mlsanity.com/paper/1706.07068/) (2017)
- [Art and the science of generative AI: A deeper dive](https://mlsanity.com/paper/2306.04141/) (2023)
- [Understanding and Creating Art with AI: Review and Outlook](https://mlsanity.com/paper/2102.09109/) (2021)
- [Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2403.11793/) (2024)
- [Can Computers Create Art?](https://mlsanity.com/paper/1801.04486/) (2018)
- ["AI enhances our performance, I have no doubt this one will do the same": The Placebo effect is robust to negative descriptions of AI](https://mlsanity.com/paper/2309.16606/) (2023)
- [Diffusion Models as Artists: Are we Closing the Gap between Humans and Machines?](https://mlsanity.com/paper/2301.11722/) (2023)
- [The Placebo Effect of Artificial Intelligence in Human-Computer Interaction](https://mlsanity.com/paper/2204.04935/) (2022)
- [Machine Learning Processes as Sources of Ambiguity: Insights from AI Art](https://mlsanity.com/paper/2403.09374/) (2024)
- [Robust One Shot Audio to Video Generation](https://mlsanity.com/paper/2012.07842/) (2020)
- [Lets Play Music: Audio-driven Performance Video Generation](https://mlsanity.com/paper/2011.02631/) (2020)
- [Secure & Personalized Music-to-Video Generation via CHARCHA](https://mlsanity.com/paper/2502.02610/) (2025)

## Built on by (most cited first, 2 of 2)

- [ExpressEdit: Fast Editing of Stylized Facial Expressions with Diffusion Models in Photoshop](https://mlsanity.com/paper/2604.03448/) (2026)
- [Bring Music The Horizon: Music-Driven 360^circ Video Generation](https://mlsanity.com/paper/2607.13471/) (2026)

## Abstract (arXiv)

Conventional music visualisation systems rely on handcrafted ad hoc transformations of shapes and colours that offer only limited expressiveness. We propose two novel pipelines for automatically generating music videos from any user-specified, vocal or instrumental song using off-the-shelf deep learning models. Inspired by the manual workflows of music video producers, we experiment on how well latent feature-based techniques can analyse audio to detect musical qualities, such as emotional cues and instrumental patterns, and distil them into textual scene descriptions using a language model. Next, we employ a generative model to produce the corresponding video clips. To assess the generated videos, we identify several critical aspects and design and conduct a preliminary user evaluation that demonstrates storytelling potential, visual coherency and emotional alignment with the music. Our findings underscore the potential of latent feature techniques and deep generative models to expand music visualisation beyond traditional approaches.