# STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment

arXiv: https://arxiv.org/abs/2409.08601 · HTML: https://mlsanity.com/paper/2409.08601/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2024.

## Built on by (most cited first, 21 of 21)

- [MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis](https://mlsanity.com/paper/2412.15322/) (2024)
- [JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization](https://mlsanity.com/paper/2503.23377/) (2025)
- [Don't Look Twice: Faster Video Transformers with Run-Length Tokenization](https://mlsanity.com/paper/2411.05222/) (2024)
- [AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation](https://mlsanity.com/paper/2412.15191/) (2024)
- [LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV](https://mlsanity.com/paper/2605.26244/) (2026)
- [FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignment](https://mlsanity.com/paper/2412.15023/) (2024)
- [DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generation](https://mlsanity.com/paper/2503.22265/) (2025)
- [3MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generation](https://mlsanity.com/paper/2511.21780/) (2025)
- [AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment](https://mlsanity.com/paper/2501.18314/) (2025)
- [Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillation](https://mlsanity.com/paper/2507.02271/) (2025)
- [Inference-Time Scaling for Joint Audio-Video Generation](https://mlsanity.com/paper/2606.03183/) (2026)
- [Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?](https://mlsanity.com/paper/2602.01623/) (2026)
- [Towards Diverse and Efficient Audio Captioning via Diffusion Models](https://mlsanity.com/paper/2409.09401/) (2024)
- [Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model](https://mlsanity.com/paper/2505.13062/) (2025)
- [Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation](https://mlsanity.com/paper/2608.04902/) (2026)
- [Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper](https://mlsanity.com/paper/2509.04957/) (2025)
- [AMAVA: Adaptive Motion-Aware Video-to-Audio Framework for Visually-Impaired Assistance](https://mlsanity.com/paper/2604.23909/) (2026)
- [MMAudio-LABEL: Audio Event Labeling via Audio Generation for Silent Video](https://mlsanity.com/paper/2605.00495/) (2026)
- [AI-Based Sound Effect Generation: A Narrative Review of Generative Models Across Input Modalities](https://mlsanity.com/paper/2608.03742/) (2026)
- [Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy](https://mlsanity.com/paper/2609.34381/) (2026)
- [AV-FlowEdit: Training-Free Prompt-Driven Audio-Visual Editing with Event-Aware Spectral Trajectory Preservation](https://mlsanity.com/paper/2610.13072/) (2026)

## Abstract (arXiv)

Visual and auditory perception are two crucial ways humans experience the world. Text-to-video generation has made remarkable progress over the past year, but the absence of harmonious audio in generated video limits its broader applications. In this paper, we propose Semantic and Temporal Aligned Video-to-Audio (STA-V2A), an approach that enhances audio generation from videos by extracting both local temporal and global semantic video features and combining these refined video features with text as cross-modal guidance. To address the issue of information redundancy in videos, we propose an onset prediction pretext task for local temporal feature extraction and an attentive pooling module for global semantic feature extraction. To supplement the insufficient semantic information in videos, we propose a Latent Diffusion Model with Text-to-Audio priors initialization and cross-modal guidance. We also introduce Audio-Audio Align, a new metric to assess audio-temporal alignment. Subjective and objective metrics demonstrate that our method surpasses existing Video-to-Audio models in generating audio with better quality, semantic consistency, and temporal alignment. The ablation experiment validated the effectiveness of each module. Audio samples are available at https://y-ren16.github.io/STAV2A.