mlsanityData to 7 Oct 2026

A Comprehensive Survey on Generative AI for Video-to-Music Generation

arXiv 2502.12489 ↗First posted 18 Feb 2025eess.AS cs.AI cs.MMcited by 4

Not on the map. Why?

Abstract (arXiv)

The burgeoning growth of video-to-music generation can be attributed to the ascendancy of multimodal generative models. However, there is a lack of literature that comprehensively combs through the work in this field. To fill this gap, this paper presents a comprehensive review of video-to-music generation using deep generative AI techniques, focusing on three key components: conditioning input construction, conditioning mechanism, and music generation frameworks. We categorize existing approaches based on their designs for each component, clarifying the roles of different strategies. Preceding this, we provide a fine-grained categorization of video and music modalities, illustrating how different categories influence the design of components within the generation pipelines. Furthermore, we summarize available multimodal datasets and evaluation metrics while highlighting ongoing challenges in the field.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 122 of its 186 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

122 of 186 references.

References · 122

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2023Segment Anythingcited by 8,158
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
38 more
  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  5. 2017The Kinetics Human Action Video Datasetcited by 1,894
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2021ViViT: A Video Vision Transformercited by 1,610
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  10. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  11. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  13. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  14. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  15. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  16. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  17. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  18. 2022High Fidelity Neural Audio Compressioncited by 673
  19. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  20. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  21. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  22. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  23. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  24. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  25. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  26. 2020Jukebox: A Generative Model for Musiccited by 535
  27. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  28. 2023Simple and Controllable Music Generationcited by 480
  29. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  30. 2023MusicLM: Generating Music From Textcited by 442
  31. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  32. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  33. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  34. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  35. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  36. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  37. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  38. 2018The Sound of Pixelscited by 258

50 most cited of 122 All 122 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Controllable Video-to-Music Generation with Multiple Time-Varying Conditionscited by 3
  2. 2025Diff-V2M: A Hierarchical Conditional Diffusion Model with Explicit Rhythmic Modeling for Video-to-Music Generationcited by 3
  3. 2026Generative AI for Video Trailer Synthesis: From Extractive Heuristics to Autoregressive Creativitycited by 2
  4. 2026NarraScore: Bridging Visual Narrative and Musical Dynamics via Hierarchical Affective Controlcited by 0