Long-Video Audio Synthesis with Multi-Agent Collaboration
Not on the map. Why?
Abstract (arXiv)
Video-to-audio synthesis, which generates synchronized audio for visual content, critically enhances viewer immersion and narrative coherence in film and interactive media. However, video-to-audio dubbing for long-form content remains an unsolved challenge due to dynamic semantic shifts, temporal misalignment, and the absence of dedicated datasets. While existing methods excel in short videos, they falter in long scenarios (e.g., movies) due to fragmented synthesis and inadequate cross-scene consistency. We propose LVAS-Agent, a novel multi-agent framework that emulates professional dubbing workflows through collaborative role specialization. Our approach decomposes long-video synthesis into four steps including scene segmentation, script generation, sound design and audio synthesis. Central innovations include a discussion-correction mechanism for scene/script refinement and a generation-retrieval loop for temporal-semantic alignment. To enable systematic evaluation, we introduce LVAS-Bench, the first benchmark with 207 professionally curated long videos spanning diverse scenarios. Experiments demonstrate superior audio-visual alignment over baseline methods. Project page: https://lvas-agent.github.io
Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 44 of its 53 references that are in the map. First posted on arXiv 13 Mar 2025.
Builds on
44 of 53 references.
References · 44
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2023Segment Anythingcited by 8,234
- 2023Visual Instruction Tuningcited by 7,869
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
32 more
- 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
- 2023ChatDev: Communicative Agents for Software Developmentcited by 919
- 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
- 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
- 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
- 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
- 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
- 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
- 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
- 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
- 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
- 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
- 2021Efficient Training of Audio Transformers with Patchoutcited by 226
- 2024Streaming Long Video Understanding with Large Language Modelscited by 167
- 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
- 2021Taming Visually Guided Sound Generationcited by 136
- 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 134
- 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
- 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
- 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
- 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
- 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
- 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
- 2020Generating Visually Aligned Sound from Videoscited by 67
- 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
- 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 53
- 2024Text-to-Audio Generation Synchronized with Videoscited by 28
- 2024Artemis: Towards Referential Understanding in Complex Videoscited by 25
- 2023ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Modelscited by 24
- 2024ChatterBox: Multi-round Multimodal Referring and Groundingcited by 24
- 2021Semantic-Aware Generation for Self-Supervised Visual Representation Learningcited by 11
- 2024Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Compositioncited by 9
Built on by
Citing papers per year: 2025: 5; 2026: 3.
8 papers.
- 2025PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generationcited by 11
- 2025Hollywood Town: Long-Video Generation via Cross-Modal Multi-Agent Orchestrationcited by 10
- 2025AutoMV: An Automatic Multi-Agent System for Music Video Generationcited by 8
- 2025EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generationcited by 6
- 2026Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Modelscited by 2
- 2026OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generationcited by 2
- 2026Agentic Visual Generation: From Generative Models to Agentic Controlcited by 1
- 2025LD-LAudio-V1: Video-to-Long-Form-Audio Generation Extension with Dual Lightweight Adapterscited by 0
- 2026Agentic Visual Generation: From Generative Models to Agentic Control6 Sep 2026
- 2026OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation16 Jun 2026
- 2026Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models24 Feb 2026
- 2025EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation31 Dec 2025
- 2025PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation30 Dec 2025
- 2025AutoMV: An Automatic Multi-Agent System for Music Video Generation13 Dec 2025
- 2025Hollywood Town: Long-Video Generation via Cross-Modal Multi-Agent Orchestration25 Oct 2025
- 2025LD-LAudio-V1: Video-to-Long-Form-Audio Generation Extension with Dual Lightweight Adapters14 Aug 2025