# Video-to-Audio Generation with Hidden Alignment

arXiv: https://arxiv.org/abs/2407.07464 · HTML: https://mlsanity.com/paper/2407.07464/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022)
- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2306.17203/) (2023)
- [Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners](https://mlsanity.com/paper/2402.17723/) (2024)
- [FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds](https://mlsanity.com/paper/2407.01494/) (2024)
- [I Hear Your True Colors: Image Guided Audio Generation](https://mlsanity.com/paper/2211.03089/) (2022)
- [Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation](https://mlsanity.com/paper/2305.18474/) (2023)
- [Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation](https://mlsanity.com/paper/2309.16429/) (2023)
- [V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models](https://mlsanity.com/paper/2308.09300/) (2023)
- [Text-to-Audio Generation Synchronized with Videos](https://mlsanity.com/paper/2403.07938/) (2024)
- [PAM: Prompting Audio-Language Models for Audio Quality Assessment](https://mlsanity.com/paper/2402.00282/) (2024)

## Built on by (most cited first, 26 of 26)

- [Video-Guided Foley Sound Generation with Multimodal Controls](https://mlsanity.com/paper/2411.17698/) (2024)
- [Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation](https://mlsanity.com/paper/2506.19774/) (2025)
- [ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing](https://mlsanity.com/paper/2506.21448/) (2025)
- [OmniAudio: Generating Spatial Audio from 360-Degree Video](https://mlsanity.com/paper/2504.14906/) (2025)
- [Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decomposition](https://mlsanity.com/paper/2503.06984/) (2025)
- [PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation](https://mlsanity.com/paper/2511.18833/) (2025)
- [FoleyBench: A Benchmark For Video-to-Audio Models](https://mlsanity.com/paper/2511.13219/) (2025)
- [EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation](https://mlsanity.com/paper/2512.24731/) (2025)
- [TA-V2A: Textually Assisted Video-to-Audio Generation](https://mlsanity.com/paper/2503.10700/) (2025)
- [AudioMoG: Guiding Audio Generation with Mixture-of-Guidance](https://mlsanity.com/paper/2509.23727/) (2025)
- [AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer](https://mlsanity.com/paper/2603.15597/) (2026)
- [ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generation](https://mlsanity.com/paper/2512.03036/) (2025)
- [Towards Diverse and Efficient Audio Captioning via Diffusion Models](https://mlsanity.com/paper/2409.09401/) (2024)
- [Video Object Segmentation-Aware Audio Generation](https://mlsanity.com/paper/2509.26604/) (2025)
- [FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scripts](https://mlsanity.com/paper/2603.19857/) (2026)
- [Benchmarking Single-Factor Physical Video-to-Audio Generation](https://mlsanity.com/paper/2605.30339/) (2026)
- [Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model](https://mlsanity.com/paper/2505.13062/) (2025)
- [Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer](https://mlsanity.com/paper/2605.30940/) (2026)
- [Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation](https://mlsanity.com/paper/2608.04902/) (2026)
- [Text Prompt is Not Enough: Sound Event Enhanced Prompt Adapter for Target Style Audio Generation](https://mlsanity.com/paper/2409.09381/) (2024)
- [MultiSoundGen: Video-to-Audio Generation for Multi-Event Scenarios via SlowFast Contrastive Audio-Visual Pretraining and Direct Preference Optimization](https://mlsanity.com/paper/2509.19999/) (2025)
- [Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation](https://mlsanity.com/paper/2510.24103/) (2025)
- [SpatialV2A: Visual-Guided High-fidelity Spatial Audio Generation](https://mlsanity.com/paper/2601.15017/) (2026)
- [SynMulti: Synthetic-to-Real Learning for Multimodal Video Understanding](https://mlsanity.com/paper/2604.12335/) (2026)
- [AMAVA: Adaptive Motion-Aware Video-to-Audio Framework for Visually-Impaired Assistance](https://mlsanity.com/paper/2604.23909/) (2026)
- [FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision](https://mlsanity.com/paper/2606.14049/) (2026)

## Abstract (arXiv)

Generating semantically and temporally aligned audio content in accordance with video input has become a focal point for researchers, particularly following the remarkable breakthrough in text-to-video generation. In this work, we aim to offer insights into the video-to-audio generation paradigm, focusing on three crucial aspects: vision encoders, auxiliary embeddings, and data augmentation techniques. Beginning with a foundational model built on a simple yet surprisingly effective intuition, we explore various vision encoders and auxiliary embeddings through ablation studies. Employing a comprehensive evaluation pipeline that emphasizes generation quality and video-audio synchronization alignment, we demonstrate that our model exhibits state-of-the-art video-to-audio generation capabilities. Furthermore, we provide critical insights into the impact of different data augmentation methods on enhancing the generation framework's overall capacity. We showcase possibilities to advance the challenge of generating synchronized audio from semantic and temporal perspectives. We hope these insights will serve as a stepping stone toward developing more realistic and accurate audio-visual generation models.