mlsanityData to 14 Oct 2026

Long-Video Audio Synthesis with Multi-Agent Collaboration

arXiv 2503.10719 ↗First posted 13 Mar 2025cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Video-to-audio synthesis, which generates synchronized audio for visual content, critically enhances viewer immersion and narrative coherence in film and interactive media. However, video-to-audio dubbing for long-form content remains an unsolved challenge due to dynamic semantic shifts, temporal misalignment, and the absence of dedicated datasets. While existing methods excel in short videos, they falter in long scenarios (e.g., movies) due to fragmented synthesis and inadequate cross-scene consistency. We propose LVAS-Agent, a novel multi-agent framework that emulates professional dubbing workflows through collaborative role specialization. Our approach decomposes long-video synthesis into four steps including scene segmentation, script generation, sound design and audio synthesis. Central innovations include a discussion-correction mechanism for scene/script refinement and a generation-retrieval loop for temporal-semantic alignment. To enable systematic evaluation, we introduce LVAS-Bench, the first benchmark with 207 professionally curated long videos spanning diverse scenarios. Experiments demonstrate superior audio-visual alignment over baseline methods. Project page: https://lvas-agent.github.io

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 44 of its 53 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

44 of 53 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2023Segment Anythingcited by 8,234
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  11. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
32 more
  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  2. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  3. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  5. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  6. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  7. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  8. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  9. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  10. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  11. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  12. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  13. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  14. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  15. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  16. 2021Taming Visually Guided Sound Generationcited by 136
  17. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 134
  18. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  19. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  20. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  21. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
  22. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  23. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  24. 2020Generating Visually Aligned Sound from Videoscited by 67
  25. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  26. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 53
  27. 2024Text-to-Audio Generation Synchronized with Videoscited by 28
  28. 2024Artemis: Towards Referential Understanding in Complex Videoscited by 25
  29. 2023ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Modelscited by 24
  30. 2024ChatterBox: Multi-round Multimodal Referring and Groundingcited by 24
  31. 2021Semantic-Aware Generation for Self-Supervised Visual Representation Learningcited by 11
  32. 2024Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Compositioncited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.