# SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing

arXiv: https://arxiv.org/abs/2605.25193 · HTML: https://mlsanity.com/paper/2605.25193/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 24 May 2026.

## Builds on (most cited first, 18 of 18)

- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks](https://mlsanity.com/paper/2403.14468/) (2024)
- [Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset](https://mlsanity.com/paper/2510.15742/) (2025)
- [InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction](https://mlsanity.com/paper/2503.20287/) (2025)
- [Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists](https://mlsanity.com/paper/2502.06734/) (2025)
- [EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning](https://mlsanity.com/paper/2509.20360/) (2025)
- [OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing](https://mlsanity.com/paper/2512.07826/) (2025)
- [HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation](https://mlsanity.com/paper/2508.16930/) (2025)
- [InstructVid2Vid: Controllable Video Editing with Natural Language Instructions](https://mlsanity.com/paper/2305.12328/) (2023)
- [In-Context Learning with Unpaired Clips for Instruction-based Video Editing](https://mlsanity.com/paper/2510.14648/) (2025)
- [StableV2V: Stablizing Shape Consistency in Video-to-Video Editing](https://mlsanity.com/paper/2411.11045/) (2024)
- [OmniV2V: Versatile Video Generation and Editing via Dynamic Content Manipulation](https://mlsanity.com/paper/2506.01801/) (2025)
- [Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoising](https://mlsanity.com/paper/2503.20782/) (2025)
- [AVI-Edit: Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner](https://mlsanity.com/paper/2512.10571/) (2025)
- [Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits](https://mlsanity.com/paper/2512.07209/) (2025)

## Built on by (most cited first, 8 of 8)

- [LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing](https://mlsanity.com/paper/2606.26740/) (2026)
- [Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing](https://mlsanity.com/paper/2606.30599/) (2026)
- [MMAE: A Massive Multitask Audio Editing Benchmark](https://mlsanity.com/paper/2606.07229/) (2026)
- [MSEditor: Toward Consistent Multi-Shot Video Editing](https://mlsanity.com/paper/2608.17559/) (2026)
- [RefVideo-6M: A Reliable Reference-Based Dataset for Instructional Video Editing](https://mlsanity.com/paper/2608.26101/) (2026)
- [TV-AudioRemover: Joint Text-Visual Guided Sound Removal with Multi-Task Hard-Mixture Curriculum](https://mlsanity.com/paper/2609.25864/) (2026)
- [AVIO: Learning to Add and Remove Sounding Objects in Audiovisual Scenes](https://mlsanity.com/paper/2609.36503/) (2026)
- [Spend Where It Matters: Particle Search with MH Refinement for Budget-Constrained Audio-Visual Alignment](https://mlsanity.com/paper/2610.12811/) (2026)

## Abstract (arXiv)

Visual and acoustic events in the physical world are inherently coupled, yet existing video editing methods typically adopt decoupled pipelines, lacking bidirectional modality interaction. This results in two key limitations: (i) audio-visual desynchronization and (ii) contextual conflicts between generated audio and preserved content. To address these, we propose SpongeBob, the first end-to-end audio-visual joint editing framework featuring bidirectional cross-modal interaction. For synchronization, a Sync-Aware Mechanism aligns visual edits with sound events via bidirectional attention, temporal alignment, and spatial constraints. For contextual consistency, a Context-Aware Module leverages acoustic and visual context attention to prevent semantic clashes. Additionally, we introduce Sync-Preserving Training and Guidance (SPTG) to enhance alignment without degrading quality. Due to the scarcity of paired data, we construct a scalable data pipeline and a large-scale subject-level dataset. We also propose SpongeBob-Bench for systematic evaluation. Experiments show SpongeBob significantly outperforms existing baselines, improving Sync-C by 30% and Ctx-F1 by 12.5%. Our project page is available at: https://hy-spongebob.github.io/.