# Generative Disco: Text-to-Video Generation for Music Visualization

arXiv: https://arxiv.org/abs/2304.08551 · HTML: https://mlsanity.com/paper/2304.08551/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 86 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 18 of 18)

- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtion](https://mlsanity.com/paper/2111.12417/) (2021)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [Sparks: Inspiration for Science Writing using Language Models](https://mlsanity.com/paper/2110.07640/) (2021)
- [NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis](https://mlsanity.com/paper/2207.09814/) (2022)
- [RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions](https://mlsanity.com/paper/2302.09466/) (2023)
- [3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows](https://mlsanity.com/paper/2210.11603/) (2022)
- [Opal: Multimodal Image Generation for News Illustration](https://mlsanity.com/paper/2204.09007/) (2022)
- [PopBlends: Strategies for Conceptual Blending with Large Language Models](https://mlsanity.com/paper/2111.04920/) (2021)
- [It's Time for Artistic Correspondence in Music and Video](https://mlsanity.com/paper/2206.07148/) (2022)
- [B-Script: Transcript-based B-roll Video Editing with Recommendations](https://mlsanity.com/paper/1902.11216/) (2019)
- [Soloist: Generating Mixed-Initiative Tutorials from Existing Guitar Instructional Videos Through Audio Processing](https://mlsanity.com/paper/2101.08846/) (2021)
- [Feel The Music: Automatically Generating A Dance For An Input Song](https://mlsanity.com/paper/2006.11905/) (2020)
- [Is there a "language of music-video clips" ? A qualitative and quantitative study](https://mlsanity.com/paper/2108.00970/) (2021)
- [Lemotif: An Affective Visual Journal Using Deep Neural Networks](https://mlsanity.com/paper/1903.07766/) (2019)

## Built on by (most cited first, 19 of 19)

- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [Diffusion Model-Based Video Editing: A Survey](https://mlsanity.com/paper/2407.07111/) (2024)
- [LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis](https://mlsanity.com/paper/2412.15214/) (2024)
- [I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors](https://mlsanity.com/paper/2305.14724/) (2023)
- [Video Diffusion Models: A Survey](https://mlsanity.com/paper/2405.03150/) (2024)
- [Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Framework](https://mlsanity.com/paper/2503.10704/) (2025)
- [Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI Workflow](https://mlsanity.com/paper/2402.09894/) (2024)
- [QVD: Post-training Quantization for Video Diffusion Models](https://mlsanity.com/paper/2407.11585/) (2024)
- [Inkspire: Supporting Design Exploration with Generative AI through Analogical Sketching](https://mlsanity.com/paper/2501.18588/) (2025)
- [Generative AI in the Construction Industry: Opportunities & Challenges](https://mlsanity.com/paper/2310.04427/) (2023)
- [DesignWeaver: Dimensional Scaffolding for Text-to-Image Product Design](https://mlsanity.com/paper/2502.09867/) (2025)
- [CoNo: Consistency Noise Injection for Tuning-free Long Video Diffusion](https://mlsanity.com/paper/2406.05082/) (2024)
- [ViPE: Visualise Pretty-much Everything](https://mlsanity.com/paper/2310.10543/) (2023)
- [Network Bending of Diffusion Models for Audio-Visual Generation](https://mlsanity.com/paper/2406.19589/) (2024)
- [Artificial Intelligence for Biomedical Video Generation](https://mlsanity.com/paper/2411.07619/) (2024)
- [Machine learning applications in archaeological practices: a review](https://mlsanity.com/paper/2501.03840/) (2025)
- [FeedQUAC: Quick Unobtrusive AI-Generated Commentary](https://mlsanity.com/paper/2504.16416/) (2025)
- [PortfolioMentor: Multimodal Generative AI Companion for Learning and Crafting Interactive Digital Art Portfolios](https://mlsanity.com/paper/2311.14091/) (2023)
- [Facilitating Longitudinal Interaction Studies of AI Systems](https://mlsanity.com/paper/2508.10252/) (2025)

## Abstract (arXiv)

Visuals can enhance our experience of music, owing to the way they can amplify the emotions and messages conveyed within it. However, creating music visualization is a complex, time-consuming, and resource-intensive process. We introduce Generative Disco, a generative AI system that helps generate music visualizations with large language models and text-to-video generation. The system helps users visualize music in intervals by finding prompts to describe the images that intervals start and end on and interpolating between them to the beat of the music. We introduce design patterns for improving these generated videos: transitions, which express shifts in color, time, subject, or style, and holds, which help focus the video on subjects. A study with professionals showed that transitions and holds were a highly expressive framework that enabled them to build coherent visual narratives. We conclude on the generalizability of these patterns and the potential of generated video for creative professionals.