# Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections

arXiv: https://arxiv.org/abs/2608.11576 · HTML: https://mlsanity.com/paper/2608.11576/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 47 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [Foley Music: Learning to Generate Music from Videos](https://mlsanity.com/paper/2007.10984/) (2020)
- [Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications](https://mlsanity.com/paper/1612.08727/) (2016)
- [VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling](https://mlsanity.com/paper/2406.04321/) (2024)
- [Quantized GAN for Complex Music Generation from Dance Videos](https://mlsanity.com/paper/2204.00604/) (2022)
- [Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model](https://mlsanity.com/paper/2311.00968/) (2023)
- [Video Background Music Generation: Dataset, Method and Evaluation](https://mlsanity.com/paper/2211.11248/) (2022)
- [M2UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models](https://mlsanity.com/paper/2311.11255/) (2023)
- [Diff-BGM: A Diffusion Model for Video Background Music Generation](https://mlsanity.com/paper/2405.11913/) (2024)
- [V2Meow: Meowing to the Visual Beat via Video-to-Music Generation](https://mlsanity.com/paper/2305.06594/) (2023)
- [VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videos](https://mlsanity.com/paper/2409.07450/) (2024)
- [MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization](https://mlsanity.com/paper/2410.12957/) (2024)
- [Content-Based Video-Music Retrieval Using Soft Intra-Modal Structure Constraint](https://mlsanity.com/paper/1704.06761/) (2017)
- [VidMusician: Video-to-Music Generation with Semantic-Rhythmic Alignment via Hierarchical Visual Features](https://mlsanity.com/paper/2412.06296/) (2024)
- [GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions](https://mlsanity.com/paper/2501.09972/) (2025)
- [Multimodal Deep Models for Predicting Affective Responses Evoked by Movies](https://mlsanity.com/paper/1909.06957/) (2019)
- [Emotion Embedding Spaces for Matching Music to Stories](https://mlsanity.com/paper/2111.13468/) (2021)
- [SONIQUE: Video Background Music Generation Using Unpaired Audio-Visual Data](https://mlsanity.com/paper/2410.03879/) (2024)
- [Predicting emotion from music videos: exploring the relative contribution of visual and auditory information to affective responses](https://mlsanity.com/paper/2202.10453/) (2022)
- [Benchmarks and leaderboards for sound demixing tasks](https://mlsanity.com/paper/2305.07489/) (2023)
- [Multimodal Music Generation with Explicit Bridges and Retrieval Augmentation](https://mlsanity.com/paper/2412.09428/) (2024)

## Built on by (most cited first, 1 of 1)

- [VIBE: Video Instruction-aligned Background music gEneration](https://mlsanity.com/paper/2608.30125/) (2026)

## Abstract (arXiv)

Video-to-music generation has drawn growing interest for its role in conveying the emotion of visual media, including film. Progress in the field, however, is hampered by a reproducibility gap: models are often trained on crawled corpora referenced through YouTube URLs that may be deleted, with the underlying data often difficult and time-consuming to retrieve. To address this, we introduce the Open Screen Soundtrack Library version 2 (OSSL-v2), a self-hosted corpus of 34,343 video clips totaling 246.4 hours, sourced from public-domain films. Unlike crawled corpora, OSSL-v2 is reproducible (i.e., not subject to link rot) and copyright-conscious, yet still large enough to train functional video-to-music models. We then use this film-domain corpus to study dialogue as a conditioning signal for video-to-music generation, motivated by the close temporal coupling between film music and on-screen speech. Specifically, we augment existing models' video cross-attention with a time axis and modulate it frame-by-frame with the dialogue track. Evaluated on both public-domain and commercial films, our approach shows improvement over the state-of-the-art baselines. The dataset is available at https://huggingface.co/datasets/McAuley-Lab/OSSL-v2.