# FilmComposer: LLM-Driven Music Production for Silent Film Clips

arXiv: https://arxiv.org/abs/2503.08147 · HTML: https://mlsanity.com/paper/2503.08147/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 46 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 30 of 37)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Movie Description](https://mlsanity.com/paper/1605.03705/) (2016)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023)
- [Foley Music: Learning to Generate Music from Videos](https://mlsanity.com/paper/2007.10984/) (2020)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [madmom: a new Python Audio and Music Signal Processing Library](https://mlsanity.com/paper/1605.07008/) (2016)
- [Video Background Music Generation with Controllable Music Transformer](https://mlsanity.com/paper/2111.08380/) (2021)
- [musicnn: Pre-trained convolutional neural networks for music audio tagging](https://mlsanity.com/paper/1909.06654/) (2019)
- [ComposerX: Multi-Agent Symbolic Music Composition with LLMs](https://mlsanity.com/paper/2404.18081/) (2024)
- [VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling](https://mlsanity.com/paper/2406.04321/) (2024)
- [Quantized GAN for Complex Music Generation from Dance Videos](https://mlsanity.com/paper/2204.00604/) (2022)
- [Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model](https://mlsanity.com/paper/2311.00968/) (2023)
- [DADAM: A Consensus-based Distributed Adaptive Gradient Method for Online Optimization](https://mlsanity.com/paper/1901.09109/) (2019)
- [Video Background Music Generation: Dataset, Method and Evaluation](https://mlsanity.com/paper/2211.11248/) (2022)
- [M2UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models](https://mlsanity.com/paper/2311.11255/) (2023)
- [Diff-BGM: A Diffusion Model for Video Background Music Generation](https://mlsanity.com/paper/2405.11913/) (2024)
- [MuseChat: A Conversational Music Recommendation System for Videos](https://mlsanity.com/paper/2310.06282/) (2023)
- [MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models](https://mlsanity.com/paper/2310.11954/) (2023)
- [AutoStudio: Crafting Consistent Subjects in Multi-turn Interactive Image Generation](https://mlsanity.com/paper/2406.01388/) (2024)
- [It's Time for Artistic Correspondence in Music and Video](https://mlsanity.com/paper/2206.07148/) (2022)

## Built on by (most cited first, 5 of 5)

- [Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing](https://mlsanity.com/paper/2604.10708/) (2026)
- [Vision-to-Music Generation: A Survey](https://mlsanity.com/paper/2503.21254/) (2025)
- [V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generation](https://mlsanity.com/paper/2603.11042/) (2026)
- [Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections](https://mlsanity.com/paper/2608.11576/) (2026)
- [JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions](https://mlsanity.com/paper/2606.01703/) (2026)

## Abstract (arXiv)

In this work, we implement music production for silent film clips using LLM-driven method. Given the strong professional demands of film music production, we propose the FilmComposer, simulating the actual workflows of professional musicians. FilmComposer is the first to combine large generative models with a multi-agent approach, leveraging the advantages of both waveform music and symbolic music generation. Additionally, FilmComposer is the first to focus on the three core elements of music production for film-audio quality, musicality, and musical development-and introduces various controls, such as rhythm, semantics, and visuals, to enhance these key aspects. Specifically, FilmComposer consists of the visual processing module, rhythm-controllable MusicGen, and multi-agent assessment, arrangement and mix. In addition, our framework can seamlessly integrate into the actual music production pipeline and allows user intervention in every step, providing strong interactivity and a high degree of creative freedom. Furthermore, we propose MusicPro-7k which includes 7,418 film clips, music, description, rhythm spots and main melody, considering the lack of a professional and high-quality film music dataset. Finally, both the standard metrics and the new specialized metrics we propose demonstrate that the music generated by our model achieves state-of-the-art performance in terms of quality, consistency with video, diversity, musicality, and musical development. Project page: https://apple-jun.github.io/FilmComposer.github.io/