mlsanityData to 7 Oct 2026

FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision

arXiv 2606.14049 ↗First posted 12 Jun 2026cs.SD cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We present FoleyGenEx, a unified video-to-audio (VTA) framework integrating multi-modal control, frame-level temporal alignment, and fine-grained semantics, enabling synchronized, versatile audio synthesis for diverse tasks. Existing VTA methods either have multi-modal control but weak temporal alignment or strong alignment but lack reference audio conditioning and semantic precision. FoleyGenEx fills this gap via three core innovations: a conditional injection mechanism for audio-controlled VTA and Foley extension, a multi-modal dynamic masking strategy preserving training synchronization, and an adverb-based data augmentation algorithm leveraging signal processing and large language models to enhance textual supervision with nuanced semantics. Experiments on AudioCaps, VGGSound, and Greatest Hits demonstrate its competitive controllable VTA performance against existing methods. Demo samples are available at https://foleygenex.github.io/FoleyGenEx.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 40 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

36 of 40 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  9. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  11. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  12. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
24 more
  1. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  2. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  3. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  4. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  5. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  6. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  7. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  8. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  9. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  10. 2015Visually Indicated Soundscited by 128
  11. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  12. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  13. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 94
  14. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  15. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  16. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  17. 2024Temporally Aligned Audio for Video with Autoregressioncited by 51
  18. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  19. 2021Audio-Visual Synchronisation in the wildcited by 39
  20. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 38
  21. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  22. 2024Taming Data and Transformers for Audio Generationcited by 28
  23. 2024Video-to-Audio Generation with Hidden Alignmentcited by 26
  24. 2024Sketch2Sound: Controllable Audio Generation via Time-Varying Signals and Sonic Imitationscited by 23

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation
  2. 2026MMControl: Unified Multi-Modal Control for Joint Audio-Video Generation
  3. 2024Video-to-Audio Generation with Fine-grained Temporal Semantics
  4. 2024FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignment
  5. 2024T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesis
  6. 2025Training-Free Multimodal Guidance for Video to Audio Generation
  7. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models
  8. 2025SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from Videos