mlsanityData to 7 Oct 2026

FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds

arXiv 2407.01494 ↗First posted 1 Jul 2024cs.CV cs.SD eess.AScited by 109

Not on the map. Why?

Abstract (arXiv)

We study Neural Foley, the automatic generation of high-quality sound effects synchronizing with videos, enabling an immersive audio-visual experience. Despite its wide range of applications, existing approaches encounter limitations when it comes to simultaneously synthesizing high-quality and video-aligned (i.e.,, semantic relevant and temporal synchronized) sounds. To overcome these limitations, we propose FoleyCrafter, a novel framework that leverages a pre-trained text-to-audio model to ensure high-quality audio generation. FoleyCrafter comprises two key components: the semantic adapter for semantic alignment and the temporal controller for precise audio-video synchronization. The semantic adapter utilizes parallel cross-attention layers to condition audio generation on video features, producing realistic sound effects that are semantically relevant to the visual content. Meanwhile, the temporal controller incorporates an onset detector and a timestampbased adapter to achieve precise audio-video alignment. One notable advantage of FoleyCrafter is its compatibility with text prompts, enabling the use of text descriptions to achieve controllable and diverse video-to-audio generation according to user intents. We conduct extensive quantitative and qualitative experiments on standard benchmarks to verify the effectiveness of FoleyCrafter. Models and codes are available at https://github.com/open-mmlab/FoleyCrafter.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

37 of 55 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  8. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  11. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  12. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
25 more
  1. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  2. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  3. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  4. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  5. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  6. 2021Taming Visually Guided Sound Generationcited by 136
  7. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 133
  8. 2015Visually Indicated Soundscited by 128
  9. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  10. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  11. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  12. 2021The Benefit Of Temporally-Strong Labels In Audio Event Classificationcited by 83
  13. 2023Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptationcited by 68
  14. 2020Generating Visually Aligned Sound from Videoscited by 67
  15. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  16. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generationcited by 54
  17. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  18. 2021Audio-Visual Synchronisation in the wildcited by 39
  19. 2024Audio-Synchronized Visual Animationcited by 32
  20. 2024Text-to-Audio Generation Synchronized with Videoscited by 28
  21. 2022Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectorscited by 25
  22. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
  23. 2023CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Modelscited by 20
  24. 2023Audio Generation with Multiple Conditional Diffusion Modelcited by 18
  25. 2024T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesiscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 54; 2026: 39.

109 papers.

Sort:

By area:

  1. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 213
  2. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  3. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  4. 2024Temporally Aligned Audio for Video with Autoregressioncited by 51
  5. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  6. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generationcited by 42
  7. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 39
  8. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 38
  9. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 35
  10. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 33
  11. 2024Video-to-Audio Generation with Hidden Alignmentcited by 26
  12. 2025AudioX: A Unified Framework for Anything-to-Audio Generationcited by 25
38 more
  1. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 22
  2. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 21
  3. 2025UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generationcited by 21
  4. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  5. 2024VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generationcited by 16
  6. 2024MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronizationcited by 14
  7. 2024LoVA: Long-form Video-to-Audio Generationcited by 13
  8. 2026Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillationcited by 13
  9. 2025JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generationcited by 12
  10. 2026OmniForcing: Unleashing Real-time Joint Audio-Visual Generationcited by 12
  11. 2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 11
  12. 2025Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decompositioncited by 10
  13. 2025TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesiscited by 10
  14. 2025PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generationcited by 10
  15. 2026EchoWM: Open and Enterable Omnimodal World Modelscited by 10
  16. 2024MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generationcited by 9
  17. 2026LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AVcited by 9
  18. 2024Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Compositioncited by 8
  19. 2025Long-Video Audio Synthesis with Multi-Agent Collaborationcited by 8
  20. 2026Omni2Sound: Towards Unified Video-Text-to-Audio Generationcited by 8
  21. 2025PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generationcited by 7
  22. 2024Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Modelscited by 6
  23. 2025FoleyBench: A Benchmark For Video-to-Audio Modelscited by 6
  24. 2026OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generationcited by 6
  25. 2024Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignmentcited by 5
  26. 2024YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controlscited by 5
  27. 2025TA-V2A: Textually Assisted Video-to-Audio Generationcited by 5
  28. 2025DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generationcited by 5
  29. 2025SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNetcited by 5
  30. 2025AudioMoG: Guiding Audio Generation with Mixture-of-Guidancecited by 5
  31. 2025EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generationcited by 5
  32. 2025AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessmentcited by 4
  33. 2025DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videoscited by 4
  34. 2025Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillationcited by 4
  35. 2025SoundReactor: Frame-level Online Video-to-Audio Generationcited by 4
  36. 2026UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gatingcited by 4
  37. 2025ReelWave: Multi-Agentic Movie Sound Generation through Multimodal LLM Conversationcited by 3
  38. 2025Extending Visual Dynamics for Video-to-Music Generationcited by 3

50 most cited of 109 All 109 (JSON)