# Efficient Video to Audio Mapper with Visual Scene Detection

arXiv: https://arxiv.org/abs/2409.09823 · HTML: https://mlsanity.com/paper/2409.09823/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2024.

## Built on by (most cited first, 4 of 4)

- [FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignment](https://mlsanity.com/paper/2412.15023/) (2024)
- [AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment](https://mlsanity.com/paper/2501.18314/) (2025)
- [ReelWave: Multi-Agentic Movie Sound Generation through Multimodal LLM Conversation](https://mlsanity.com/paper/2503.07217/) (2025)
- [Gotta Hear Them All: Towards Sound Source Aware Audio Generation](https://mlsanity.com/paper/2411.15447/) (2024)

## Abstract (arXiv)

Video-to-audio (V2A) generation aims to produce corresponding audio given silent video inputs. This task is particularly challenging due to the cross-modality and sequential nature of the audio-visual features involved. Recent works have made significant progress in bridging the domain gap between video and audio, generating audio that is semantically aligned with the video content. However, a critical limitation of these approaches is their inability to effectively recognize and handle multiple scenes within a video, often leading to suboptimal audio generation in such cases. In this paper, we first reimplement a state-of-the-art V2A model with a slightly modified light-weight architecture, achieving results that outperform the baseline. We then propose an improved V2A model that incorporates a scene detector to address the challenge of switching between multiple visual scenes. Results on VGGSound show that our model can recognize and handle multiple scenes within a video and achieve superior performance against the baseline for both fidelity and relevance.