mlsanityData to 7 Oct 2026

VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling

arXiv 2406.04321 ↗First posted 6 Jun 2024cs.CV cs.LG cs.MMcited by 39

Not on the map. Why?

Abstract (arXiv)

In this work, we systematically study music generation conditioned solely on the video. First, we present a large-scale dataset comprising 360K video-music pairs, including various genres such as movie trailers, advertisements, and documentaries. Furthermore, we propose VidMuse, a simple framework for generating music aligned with video inputs. VidMuse stands out by producing high-fidelity music that is both acoustically and semantically aligned with the video. By incorporating local and global visual cues, VidMuse enables the creation of musically coherent audio tracks that consistently match the video content through Long-Short-Term modeling. Through extensive experiments, VidMuse outperforms existing models in terms of audio quality, diversity, and audio-visual alignment. The code and datasets are available at https://vidmuse.github.io/.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 88 of its 107 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

88 of 107 references.

References · 88

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2018SlowFast Networks for Video Recognitioncited by 1,862
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  4. 2021ViViT: A Video Vision Transformercited by 1,610
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  7. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  10. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  11. 2021Video Swin Transformercited by 1,090
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  13. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  14. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  15. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  16. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  17. 2022High Fidelity Neural Audio Compressioncited by 673
  18. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  19. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  20. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  21. 2020Jukebox: A Generative Model for Musiccited by 535
  22. 2023Simple and Controllable Music Generationcited by 480
  23. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  24. 2023MusicLM: Generating Music From Textcited by 442
  25. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  26. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  27. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  28. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  29. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  30. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  31. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  32. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  33. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  34. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  35. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 221
  36. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 215
  37. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  38. 2021Symbolic Music Generation with Diffusion Modelscited by 152

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 19; 2026: 15.

39 papers.

Sort:

By area:

  1. 2024Foundation Models for Music: A Surveycited by 34
  2. 2024Taming Data and Transformers for Audio Generationcited by 28
  3. 2025AudioX: A Unified Framework for Anything-to-Audio Generationcited by 25
  4. 2025Generative AI for Cel-Animation: A Surveycited by 17
  5. 2025Generative AI for Film Creation: A Survey of Recent Advancescited by 17
  6. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  7. 2024MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronizationcited by 14
  8. 2025HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronizationcited by 14
  9. 2024MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptionscited by 13
  10. 2025Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoisingcited by 13
  11. 2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 11
  12. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 10
27 more
  1. 2025AutoMV: An Automatic Multi-Agent System for Music Video Generationcited by 7
  2. 2024Multimodal Music Generation with Explicit Bridges and Retrieval Augmentationcited by 6
  3. 2026Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generationcited by 6
  4. 2025FilmComposer: LLM-Driven Music Production for Silent Film Clipscited by 5
  5. 2025DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generationcited by 5
  6. 2025A Comprehensive Survey on Generative AI for Video-to-Music Generationcited by 4
  7. 2025Vision-to-Music Generation: A Surveycited by 4
  8. 2025Automatic Stage Lighting Control: Is it a Rule-Driven Process or Generative Task?cited by 4
  9. 2025UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoEcited by 4
  10. 2025Extending Visual Dynamics for Video-to-Music Generationcited by 3
  11. 2025Bob's Confetti: Phonetic Memorization Attacks in Music and Video Generationcited by 3
  12. 2025Controllable Video-to-Music Generation with Multiple Time-Varying Conditionscited by 3
  13. 2026V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generationcited by 2
  14. 2026Generative AI for Video Trailer Synthesis: From Extractive Heuristics to Autoregressive Creativitycited by 2
  15. 2026Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generationcited by 2
  16. 2025Cross-Modal Learning for Music-to-Music-Video Description Generationcited by 1
  17. 2026VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categoriescited by 1
  18. 2026BEAT: Rhythm-Elastic Alignment for Agentic Music-guided Movie Trailer Generationcited by 1
  19. 2026AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generationcited by 1
  20. 2026Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collectionscited by 1
  21. 2025EXPOTION: Facial Expression and Motion Control for Multimodal Music Generationcited by 0
  22. 2026SyncTrack: Rhythmic Stability and Synchronization in Multi-Track Music Generationcited by 0
  23. 2026JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitionscited by 0
  24. 2026AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Modelscited by 0
  25. 2026HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusioncited by 0
  26. 2026VIBE: Video Instruction-aligned Background music gEnerationcited by 0
  27. 2026Video-to-Music Generation for Gameplay Videoscited by 0