# Semantically consistent Video-to-Audio Generation using Multimodal Language Large Model

arXiv: https://arxiv.org/abs/2404.16305 · HTML: https://mlsanity.com/paper/2404.16305/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2024.

## Built on by (most cited first, 11 of 11)

- [Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation](https://mlsanity.com/paper/2510.01284/) (2025)
- [KeyVideoLLM: Towards Large-scale Video Keyframe Selection](https://mlsanity.com/paper/2407.03104/) (2024)
- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation](https://mlsanity.com/paper/2412.15191/) (2024)
- [Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoising](https://mlsanity.com/paper/2503.20782/) (2025)
- [FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignment](https://mlsanity.com/paper/2412.15023/) (2024)
- [AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment](https://mlsanity.com/paper/2501.18314/) (2025)
- [Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation](https://mlsanity.com/paper/2507.04959/) (2025)
- [MAJORScore: A Novel Metric for Evaluating Multimodal Relevance via Joint Representation](https://mlsanity.com/paper/2509.21365/) (2025)
- [FoleyGRAM: Video-to-Audio Generation with GRAM-Aligned Multimodal Encoders](https://mlsanity.com/paper/2510.05829/) (2025)
- [AMAVA: Adaptive Motion-Aware Video-to-Audio Framework for Visually-Impaired Assistance](https://mlsanity.com/paper/2604.23909/) (2026)

## Abstract (arXiv)

Existing works have made strides in video generation, but the lack of sound effects (SFX) and background music (BGM) hinders a complete and immersive viewer experience. We introduce a novel semantically consistent v ideo-to-audio generation framework, namely SVA, which automatically generates audio semantically consistent with the given video content. The framework harnesses the power of multimodal large language model (MLLM) to understand video semantics from a key frame and generate creative audio schemes, which are then utilized as prompts for text-to-audio models, resulting in video-to-audio generation with natural language as an interface. We show the satisfactory performance of SVA through case study and discuss the limitations along with the future research direction. The project page is available at https://huiz-a.github.io/audio4video.github.io/.