mlsanityData to 14 Oct 2026

HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronization

arXiv 2503.01725 ↗First posted 3 Mar 2025cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

This paper introduces HarmonySet, a comprehensive dataset designed to advance video-music understanding. HarmonySet consists of 48,328 diverse video-music pairs, annotated with detailed information on rhythmic synchronization, emotional alignment, thematic coherence, and cultural relevance. We propose a multi-step human-machine collaborative framework for efficient annotation, combining human insights with machine-generated descriptions to identify key transitions and assess alignment across multiple dimensions. Additionally, we introduce a novel evaluation framework with tasks and metrics to assess the multi-dimensional alignment of video and music, including rhythm, emotion, theme, and cultural context. Our extensive experiments demonstrate that HarmonySet, along with the proposed evaluation framework, significantly improves the ability of multimodal models to capture and analyze the intricate relationships between video and music.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 57 of its 77 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

57 of 77 references.

References · 57

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2017The Kinetics Human Action Video Datasetcited by 1,909
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
38 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  4. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  5. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  6. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  9. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  10. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  11. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  12. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  13. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  14. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 485
  15. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  16. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  17. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  18. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  19. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  20. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  21. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  22. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  23. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  24. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  25. 2016Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applicationscited by 61
  26. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 55
  27. 2023FoleyGen: Visually-Guided Audio Generationcited by 40
  28. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 40
  29. 2022Quantized GAN for Complex Music Generation from Dance Videoscited by 37
  30. 2024Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrievalcited by 36
  31. 2023FunQA: Towards Surprising Video Comprehensioncited by 35
  32. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  33. 2022Video Background Music Generation: Dataset, Method and Evaluationcited by 29
  34. 2024Diff-BGM: A Diffusion Model for Video Background Music Generationcited by 23
  35. 2023MuseChat: A Conversational Music Recommendation System for Videoscited by 21
  36. 2022It's Time for Artistic Correspondence in Music and Videocited by 18
  37. 2023Language-Guided Music Recommendation for Video via Prompt Analogiescited by 18
  38. 2023Long-Term Rhythmic Video Soundtrackercited by 17

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 9.

14 papers.

Sort:

By area:

  1. 2025AudioX: A Unified Framework for Anything-to-Audio Generationcited by 27
  2. 2025FlexSelect: Flexible Token Selection for Efficient Long Video Understandingcited by 18
  3. 2025From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignmentcited by 6
  4. 2025Vision-to-Music Generation: A Surveycited by 4
  5. 2025MVAD: A Benchmark Dataset for Multimodal AI-Generated Video-Audio Detectioncited by 3
  6. 2026V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generationcited by 2
  7. 2026Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generationcited by 2
  8. 2026Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videoscited by 2
  9. 2026VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categoriescited by 1
  10. 2026AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generationcited by 1
  11. 2026JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitionscited by 0
  12. 2026Empowering Long-form Omni-modal Understanding with Robust Audio Perceptioncited by 0
2 more
  1. 2026A Survey of Large Models in Sportscited by 0
  2. 2026VIBE: Video Instruction-aligned Background music gEnerationcited by 0