# V2Meow: Meowing to the Visual Beat via Video-to-Music Generation

arXiv: https://arxiv.org/abs/2305.06594 · HTML: https://mlsanity.com/paper/2305.06594/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 55 references that are in the map.
- First posted on arXiv 11 May 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [Diffsound: Discrete Diffusion Model for Text-to-sound Generation](https://mlsanity.com/paper/2207.09983/) (2022)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023)
- [MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment](https://mlsanity.com/paper/1709.06298/) (2017)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Visually Indicated Sounds](https://mlsanity.com/paper/1512.08512/) (2015)
- [Visual to Sound: Generating Natural Sound for Videos in the Wild](https://mlsanity.com/paper/1712.01393/) (2017)
- [MuLan: A Joint Embedding of Music Audio and Natural Language](https://mlsanity.com/paper/2208.12415/) (2022)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [Foley Music: Learning to Generate Music from Videos](https://mlsanity.com/paper/2007.10984/) (2020)
- [LEAF: A Learnable Frontend for Audio Classification](https://mlsanity.com/paper/2101.08596/) (2021)
- [This Time with Feeling: Learning Expressive Musical Performance](https://mlsanity.com/paper/1808.03715/) (2018)
- [Generating Visually Aligned Sound from Videos](https://mlsanity.com/paper/2008.00820/) (2020)
- [Deep Cross-Modal Audio-Visual Generation](https://mlsanity.com/paper/1704.08292/) (2017)

## Built on by (most cited first, 18 of 18)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Generative AI for Cel-Animation: A Survey](https://mlsanity.com/paper/2501.06250/) (2025)
- [VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videos](https://mlsanity.com/paper/2409.07450/) (2024)
- [MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization](https://mlsanity.com/paper/2410.12957/) (2024)
- [Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Composition](https://mlsanity.com/paper/2410.03335/) (2024)
- [AutoMV: An Automatic Multi-Agent System for Music Video Generation](https://mlsanity.com/paper/2512.12196/) (2025)
- [A Comprehensive Survey on Generative AI for Video-to-Music Generation](https://mlsanity.com/paper/2502.12489/) (2025)
- [Vision-to-Music Generation: A Survey](https://mlsanity.com/paper/2503.21254/) (2025)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)
- [SummDiff: Generative Modeling of Video Summarization with Diffusion](https://mlsanity.com/paper/2510.08458/) (2025)
- [Controllable Video-to-Music Generation with Multiple Time-Varying Conditions](https://mlsanity.com/paper/2507.20627/) (2025)
- [Diffusion-based Symbolic Music Generation with Structured State Space Models](https://mlsanity.com/paper/2507.20128/) (2025)
- [Prevailing Research Areas for Music AI in the Era of Foundation Models](https://mlsanity.com/paper/2409.09378/) (2024)
- [Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections](https://mlsanity.com/paper/2608.11576/) (2026)
- [EXPOTION: Facial Expression and Motion Control for Multimodal Music Generation](https://mlsanity.com/paper/2507.04955/) (2025)
- [NarraScore: Bridging Visual Narrative and Musical Dynamics via Hierarchical Affective Control](https://mlsanity.com/paper/2602.09070/) (2026)
- [JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions](https://mlsanity.com/paper/2606.01703/) (2026)
- [Video-to-Music Generation for Gameplay Videos](https://mlsanity.com/paper/2609.31810/) (2026)

## Abstract (arXiv)

Video-to-music generation demands both a temporally localized high-quality listening experience and globally aligned video-acoustic signatures. While recent music generation models excel at the former through advanced audio codecs, the exploration of video-acoustic signatures has been confined to specific visual scenarios. In contrast, our research confronts the challenge of learning globally aligned signatures between video and music directly from paired music and videos, without explicitly modeling domain-specific rhythmic or semantic relationships. We propose V2Meow, a video-to-music generation system capable of producing high-quality music audio for a diverse range of video input types using a multi-stage autoregressive model. Trained on 5k hours of music audio clips paired with video frames mined from in-the-wild music videos, V2Meow is competitive with previous domain-specific models when evaluated in a zero-shot manner. It synthesizes high-fidelity music audio waveforms solely by conditioning on pre-trained general-purpose visual features extracted from video frames, with optional style control via text prompts. Through both qualitative and quantitative evaluations, we demonstrate that our model outperforms various existing music generation systems in terms of visual-audio correspondence and audio quality. Music samples are available at tinyurl.com/v2meow.