mlsanityData to 14 Oct 2026

Video Background Music Generation with Controllable Music Transformer

arXiv 2111.08380 ↗First posted 16 Nov 2021cs.MM cs.SD eess.AScited by 56

Not on the map. Why?

Abstract (arXiv)

In this work, we address the task of video background music generation. Some previous works achieve effective music generation but are unable to generate melodious music tailored to a particular video, and none of them considers the video-music rhythmic consistency. To generate the background music that matches the given video, we first establish the rhythmic relations between video and background music. In particular, we connect timing, motion speed, and motion saliency from video with beat, simu-note density, and simu-note strength from music, respectively. We then propose CMT, a Controllable Music Transformer that enables local control of the aforementioned rhythmic features and global control of the music genre and instruments. Objective and subjective evaluations show that the generated background music has achieved satisfactory compatibility with the input videos, and at the same time, impressive music quality. Code and models are available at https://github.com/wzk1015/video-bgm-generation.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Nov 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 15; 2024: 11; 2025: 12; 2026: 10.

56 papers.

Sort:

By area:

  1. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 227
  2. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  3. 2023MuseCoco: Generating Symbolic Music from Textcited by 52
  4. 2022Discrete Contrastive Diffusion for Cross-Modal Music and Image Generationcited by 44
  5. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  6. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 40
  7. 2024M3GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generationcited by 38
  8. 2022Quantized GAN for Complex Music Generation from Dance Videoscited by 37
  9. 2023Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer modelcited by 37
  10. 2024ITINERA: Integrating Spatial Optimization with Large Language Models for Open-domain Urban Itinerary Planningcited by 33
  11. 2022Video Background Music Generation: Dataset, Method and Evaluationcited by 29
  12. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 29
38 more
  1. 2025AudioX: A Unified Framework for Anything-to-Audio Generationcited by 27
  2. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
  3. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  4. 2023V2Meow: Meowing to the Visual Beat via Video-to-Music Generationcited by 18
  5. 2023XVO: Generalized Visual Odometry via Cross-Modal Self-Trainingcited by 18
  6. 2023Long-Term Rhythmic Video Soundtrackercited by 17
  7. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  8. 2023Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generationcited by 15
  9. 2024VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videoscited by 15
  10. 2024MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronizationcited by 14
  11. 2023MelodyGLM: Multi-task Pre-training for Symbolic Melody Generationcited by 10
  12. 2024Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Surveycited by 10
  13. 2023A Survey of AI Music Generation Tools and Modelscited by 9
  14. 2022ComMU: Dataset for Combinatorial Music Generationcited by 8
  15. 2023Regeneration Learning: A Learning Paradigm for Data Generationcited by 8
  16. 2023CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modelingcited by 7
  17. 2024Multimodal Music Generation with Explicit Bridges and Retrieval Augmentationcited by 6
  18. 2023WuYun: Exploring hierarchical skeleton-guided melody generation using knowledge-enhanced deep learningcited by 5
  19. 2023EmoGen: Eliminating Subjective Bias in Emotional Music Generationcited by 5
  20. 2024A Survey of Deep Learning Audio Generation Methodscited by 5
  21. 2024FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignmentcited by 5
  22. 2025FilmComposer: LLM-Driven Music Production for Silent Film Clipscited by 5
  23. 2025Enhancing Dance-to-Music Generation via Negative Conditioning Latent Diffusion Modelcited by 5
  24. 2024The NES Video-Music Database: A Dataset of Symbolic Video Game Music Paired with Gameplay Videoscited by 4
  25. 2025A Comprehensive Survey on Generative AI for Video-to-Music Generationcited by 4
  26. 2025Vision-to-Music Generation: A Surveycited by 4
  27. 2025A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectivescited by 4
  28. 2025Extending Visual Dynamics for Video-to-Music Generationcited by 4
  29. 2025MusFlow: Multimodal Music Generation via Conditional Flow Matchingcited by 4
  30. 2025Controllable Video-to-Music Generation with Multiple Time-Varying Conditionscited by 3
  31. 2025Video Soundtrack Generation by Aligning Emotions and Temporal Boundariescited by 2
  32. 2026V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generationcited by 2
  33. 2026Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generationcited by 2
  34. 2022FIGARO: Generating Symbolic Music with Fine-Grained Artistic Controlcited by 1
  35. 2026VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categoriescited by 1
  36. 2026AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generationcited by 1
  37. 2025MultiSoundGen: Video-to-Audio Generation for Multi-Event Scenarios via SlowFast Contrastive Audio-Visual Pretraining and Direct Preference Optimizationcited by 0
  38. 2025Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approachcited by 0

50 most cited of 56 All 56 (JSON)