mlsanityData to 7 Oct 2026

LoVA: Long-form Video-to-Audio Generation

arXiv 2409.15157 ↗First posted 23 Sep 2024cs.SD cs.MM eess.AScited by 13

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Video-to-audio (V2A) generation is important for video editing and post-processing, enabling the creation of semantics-aligned audio for silent video. However, most existing methods focus on generating short-form audio for short video segment (less than 10 seconds), while giving little attention to the scenario of long-form video inputs. For current UNet-based diffusion V2A models, an inevitable problem when handling long-form audio generation is the inconsistencies within the final concatenated audio. In this paper, we first highlight the importance of long-form V2A problem. Besides, we propose LoVA, a novel model for Long-form Video-to-Audio generation. Based on the Diffusion Transformer (DiT) architecture, LoVA proves to be more effective at generating long-form audio compared to existing autoregressive models and UNet-based diffusion models. Extensive objective and subjective experiments demonstrate that LoVA achieves comparable performance on 10-second V2A benchmark and outperforms all other baselines on a benchmark with long-form video input.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 27 of its 32 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

27 of 32 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Long-form music generation with latent diffusion16 Apr
  2. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  3. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models29 Jun
  4. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation29 May
  5. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  8. 2016Improved Techniques for Training GANscited by 2,938
  9. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  10. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 954
  11. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  12. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
10 more
  1. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  2. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  3. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 221
  4. 2024Stable Audio Opencited by 192
  5. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  6. 2021Taming Visually Guided Sound Generationcited by 136
  7. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  8. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  9. 2023Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baselinecited by 45
  10. 2023FoleyGen: Visually-Guided Audio Generationcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 21
  2. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  3. 2025Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interactioncited by 9
  4. 2025FoleyBench: A Benchmark For Video-to-Audio Modelscited by 6
  5. 2025VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learningcited by 3
  6. 2025ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generationcited by 3
  7. 2026Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Modelscited by 1
  8. 2026SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learningcited by 1
  9. 2025LD-LAudio-V1: Video-to-Long-Form-Audio Generation Extension with Dual Lightweight Adapterscited by 0
  10. 2025MeanFlow-Accelerated Multimodal Video-to-Audio Synthesis via One-Step Generationcited by 0
  11. 2025SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from Videoscited by 0
  12. 2026Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Modelscited by 0
1 more
  1. 2026AI-Based Sound Effect Generation: A Narrative Review of Generative Models Across Input Modalitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generation
  2. 2026Diffusion Models for Joint Audio-Video Generation
  3. 2023The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion
  4. 2025StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation
  5. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation
  6. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching
  7. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
  8. 20253MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generation