mlsanityData to 7 Oct 2026

Temporally Aligned Audio for Video with Autoregression

arXiv 2409.13689 ↗First posted 20 Sep 2024cs.CV cs.MM cs.SDcited by 51

Not on the map. Why?

Abstract (arXiv)

We introduce V-AURA, the first autoregressive model to achieve high temporal alignment and relevance in video-to-audio generation. V-AURA uses a high-framerate visual feature extractor and a cross-modal audio-visual feature fusion strategy to capture fine-grained visual motion events and ensure precise temporal alignment. Additionally, we propose VisualSound, a benchmark dataset with high audio-visual relevance. VisualSound is based on VGGSound, a video dataset consisting of in-the-wild samples extracted from YouTube. During the curation, we remove samples where auditory events are not aligned with the visual ones. V-AURA outperforms current state-of-the-art models in temporal alignment and semantic relevance while maintaining comparable audio quality. Code, samples, VisualSound and models are available at https://v-aura.notion.site

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 20 Sep 2024.

Builds on

28 of 33 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  8. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  9. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  10. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  11. 2023Simple and Controllable Music Generationcited by 480
  12. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
16 more
  1. 2022AudioGen: Textually Guided Audio Generationcited by 292
  2. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  3. 2021Taming Visually Guided Sound Generationcited by 136
  4. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 133
  5. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  6. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  7. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 94
  8. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  9. 2020Generating Visually Aligned Sound from Videoscited by 67
  10. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  11. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  12. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  13. 2023FoleyGen: Visually-Guided Audio Generationcited by 40
  14. 2022Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectorscited by 25
  15. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
  16. 2021FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videoscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 32; 2026: 15.

51 papers.

Sort:

By area:

  1. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  2. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  3. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 38
  4. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 33
  5. 2025KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generationcited by 24
  6. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 21
  7. 2025UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generationcited by 21
  8. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  9. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  10. 2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 11
  11. 2025VGGSounder: Audio-Visual Evaluations for Foundation Modelscited by 8
  12. 2026Omni2Sound: Towards Unified Video-Text-to-Audio Generationcited by 8
38 more
  1. 2025Syncphony: Synchronized Audio-to-Video Generation with Diffusion Transformerscited by 6
  2. 2025FoleyBench: A Benchmark For Video-to-Audio Modelscited by 6
  3. 2026Native Audio-Visual Alignment for Generationcited by 6
  4. 2024YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controlscited by 5
  5. 2025SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNetcited by 5
  6. 2025UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalitiescited by 5
  7. 2025DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videoscited by 4
  8. 2025Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillationcited by 4
  9. 2025SoundReactor: Frame-level Online Video-to-Audio Generationcited by 4
  10. 2025VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learningcited by 3
  11. 2025MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separationcited by 3
  12. 2025Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Videocited by 3
  13. 2025Hear What Matters! Text-conditioned Selective Video-to-Audio Generationcited by 3
  14. 2025ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generationcited by 3
  15. 2025LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Modelscited by 2
  16. 2025Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidancecited by 2
  17. 2025Video Object Segmentation-Aware Audio Generationcited by 2
  18. 2026AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfercited by 2
  19. 2026FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scriptscited by 2
  20. 2026WavFlow: Audio Generation in Waveform Spacecited by 2
  21. 2025PAVAS: Physics-Aware Video-to-Audio Synthesiscited by 1
  22. 2026Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Modelscited by 1
  23. 2026VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categoriescited by 1
  24. 2026Benchmarking Single-Factor Physical Video-to-Audio Generationcited by 1
  25. 2026AVTok: 1D Unified Tokenization for Holistic Audio-Video Generationcited by 1
  26. 2025Enhance Generation Quality of Flow Matching V2A Model via Multi-Step CoT-Like Guidance and Combined Preference Optimizationcited by 0
  27. 2025Towards Video to Piano Music Generation with Chain-of-Perform Support Benchmarkscited by 0
  28. 2025EXPOTION: Facial Expression and Motion Control for Multimodal Music Generationcited by 0
  29. 2025LD-LAudio-V1: Video-to-Long-Form-Audio Generation Extension with Dual Lightweight Adapterscited by 0
  30. 2025Efficient Video-to-Audio Generation via Multiple Foundation Models Mappercited by 0
  31. 2025MultiSoundGen: Video-to-Audio Generation for Multi-Event Scenarios via SlowFast Contrastive Audio-Visual Pretraining and Direct Preference Optimizationcited by 0
  32. 2025SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from Videoscited by 0
  33. 2025Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generationcited by 0
  34. 2026SpatialV2A: Visual-Guided High-fidelity Spatial Audio Generationcited by 0
  35. 2026Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformercited by 0
  36. 2026FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precisioncited by 0
  37. 2026HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusioncited by 0
  38. 2026Passing: An Endless Journey through Reconstructed Spacetime with AI-Generated Soundcited by 0

50 most cited of 51 All 51 (JSON)