mlsanityData to 14 Oct 2026

Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

arXiv 2306.17203 ↗First posted 29 Jun 2023cs.SD cs.CV cs.LGcited by 134

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

The Video-to-Audio (V2A) model has recently gained attention for its practical application in generating audio directly from silent videos, particularly in video/film production. However, previous methods in V2A have limited generation quality in terms of temporal synchronization and audio-visual relevance. We present Diff-Foley, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) that generates high-quality audio with improved synchronization and audio-visual relevance. We adopt contrastive audio-visual pretraining (CAVP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. The CAVP-aligned features enable LDM to capture the subtler audio-visual correlation via a cross-attention module. We further significantly improve sample quality with `double guidance'. Diff-Foley achieves state-of-the-art V2A performance on current large scale V2A dataset. Furthermore, we demonstrate Diff-Foley practical applicability and generalization capabilities via downstream finetuning. Project Page: see https://diff-foley.github.io/

Key facts. Cited by 134 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 46 of its 52 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

46 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  2. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
31 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  3. 2022Classifier-Free Diffusion Guidancecited by 5,847
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  8. 2022Video Diffusion Modelscited by 2,305
  9. 2017The Kinetics Human Action Video Datasetcited by 1,909
  10. 2018SlowFast Networks for Video Recognitioncited by 1,867
  11. 2017Decoupled Weight Decay Regularizationcited by 1,831
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  13. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  14. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  15. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  16. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
  17. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  18. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  19. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  20. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  21. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  22. 2022AudioGen: Textually Guided Audio Generationcited by 295
  23. 2021Taming Visually Guided Sound Generationcited by 136
  24. 2015Visually Indicated Soundscited by 128
  25. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  26. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  27. 2020Generating Visually Aligned Sound from Videoscited by 67
  28. 2017Vid2speech: Speech Reconstruction from Silent Videocited by 53
  29. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  30. 2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
  31. 2021FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videoscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 45; 2025: 50; 2026: 36.

134 papers.

Sort:

By area:

  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  2. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 236
  3. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  4. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  5. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
  6. 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 83
  7. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  8. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 53
  9. 2024Temporally Aligned Audio for Video with Autoregressioncited by 52
  10. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  11. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  12. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 43
38 more
  1. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 40
  2. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 38
  3. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  4. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  5. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 34
  6. 2024Audio-Synchronized Visual Animationcited by 32
  7. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  8. 2025AudioX: A Unified Framework for Anything-to-Audio Generationcited by 27
  9. 2024Video-to-Audio Generation with Hidden Alignmentcited by 26
  10. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 25
  11. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
  12. 2024MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generationcited by 23
  13. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
  14. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 22
  15. 2025UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generationcited by 22
  16. 2024Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videoscited by 21
  17. 2024A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generationcited by 19
  18. 2024Language-Guided Joint Audio-Visual Editing via One-Shot Adaptationcited by 19
  19. 2024A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generationcited by 18
  20. 2025Generative AI for Cel-Animation: A Surveycited by 17
  21. 2024Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Soundcited by 16
  22. 2024VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generationcited by 16
  23. 2026OmniForcing: Unleashing Real-time Joint Audio-Visual Generationcited by 16
  24. 2024VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videoscited by 15
  25. 2024MusicFlow: Cascaded Flow Matching for Text Guided Music Generationcited by 15
  26. 2024MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronizationcited by 14
  27. 2025Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoisingcited by 14
  28. 2024PEAVS: Perceptual Evaluation of Audio-Visual Synchrony Grounded in Viewers' Opinion Scorescited by 13
  29. 2024Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Modelscited by 13
  30. 2024LoVA: Long-form Video-to-Audio Generationcited by 13
  31. 2026EchoWM: Open and Enterable Omnimodal World Modelscited by 13
  32. 2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 12
  33. 2024DISCO: Efficient Diffusion Solver for Large-Scale Combinatorial Optimization Problemscited by 11
  34. 2025ViSAGe: Video-to-Spatial Audio Generationcited by 11
  35. 2026Omni2Sound: Towards Unified Video-Text-to-Audio Generationcited by 11
  36. 2024SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Soundcited by 10
  37. 2025Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decompositioncited by 10
  38. 2025TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesiscited by 10

50 most cited of 134 All 134 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation
  2. 2025DreamFoley: Scalable VLMs for High-Fidelity Video-to-Audio Generation
  3. 2026Diffusion Models for Joint Audio-Video Generation
  4. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation
  5. 2024STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment
  6. 2023AADiff: Audio-Aligned Video Synthesis with Text-to-Image Diffusion
  7. 2025Syncphony: Synchronized Audio-to-Video Generation with Diffusion Transformers
  8. 2026Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention