mlsanityData to 7 Oct 2026

FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos

arXiv 2107.09262 ↗First posted 20 Jul 2021cs.LG cs.AI cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Deep learning based visual to sound generation systems essentially need to be developed particularly considering the synchronicity aspects of visual and audio features with time. In this research we introduce a novel task of guiding a class conditioned generative adversarial network with the temporal visual information of a video input for visual to sound generation task adapting the synchronicity traits between audio-visual modalities. Our proposed FoleyGAN model is capable of conditioning action sequences of visual events leading towards generating visually aligned realistic sound tracks. We expand our previously proposed Automatic Foley dataset to train with FoleyGAN and evaluate our synthesized sound through human survey that shows noteworthy (on average 81%) audio-visual synchronicity performance. Our approach also outperforms in statistical experiments compared with other baseline models and audio-visual datasets.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 36 of its 69 references that are in the map. First posted on arXiv 20 Jul 2021.

Builds on

36 of 69 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2016Improved Techniques for Training GANscited by 2,938
  7. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  9. 2018Self-Attention Generative Adversarial Networkscited by 809
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  11. 2017Look, Listen and Learncited by 452
  12. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
24 more
  1. 2016Adversarial Feature Learningcited by 350
  2. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  3. 2017Temporal Relational Reasoning in Videoscited by 324
  4. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  5. 2019Large Scale Adversarial Representation Learningcited by 238
  6. 2018Adversarial Audio Synthesiscited by 222
  7. 2016Adversarially Learned Inferencecited by 216
  8. 2015Visually Indicated Soundscited by 128
  9. 2019GANSynth: Adversarial Neural Audio Synthesiscited by 114
  10. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  11. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  12. 2017MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generationcited by 72
  13. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  14. 2020Generating Visually Aligned Sound from Videoscited by 67
  15. 2017Deep Cross-Modal Audio-Visual Generationcited by 63
  16. 20182.5D Visual Soundcited by 63
  17. 2018On GANs and GMMscited by 62
  18. 2019MelNet: A Generative Model for Audio in the Frequency Domaincited by 41
  19. 2019Vision-Infused Deep Audio Inpaintingcited by 40
  20. 2018Scene-Aware Audio for 360textdegree Videoscited by 25
  21. 2020AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos with Deep Learningcited by 19
  22. 2017AENet: Learning Deep Audio Features for Video Analysiscited by 13
  23. 2019Adversarial Generation of Time-Frequency Features with application in audio synthesiscited by 12
  24. 2020High-Fidelity Audio Generation and Representation Learning with Guided Adversarial Autoencodercited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 4; 2024: 7; 2025: 3; 2026: 3.

20 papers.

Sort:

By area:

  1. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 133
  2. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  3. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 52
  4. 2024Temporally Aligned Audio for Video with Autoregressioncited by 51
  5. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
  6. 2024T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesiscited by 17
  7. 2024A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generationcited by 17
  8. 2024Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Soundcited by 16
  9. 2024MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronizationcited by 14
  10. 2022VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglementcited by 11
  11. 2022A Proposal for Foley Sound Synthesis Challengecited by 8
  12. 2021Soundify: Matching Sound Effects to Videocited by 6
8 more
  1. 2024FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignmentcited by 5
  2. 2025TA-V2A: Textually Assisted Video-to-Audio Generationcited by 5
  3. 2023A Demand-Driven Perspective on Generative Audio AIcited by 4
  4. 2025StereoSync: Spatially-Aware Stereo Audio Generation from Videocited by 0
  5. 2025FoleyGRAM: Video-to-Audio Generation with GRAM-Aligned Multimodal Encoderscited by 0
  6. 2026AMAVA: Adaptive Motion-Aware Video-to-Audio Framework for Visually-Impaired Assistancecited by 0
  7. 2026Exploring Efficient Waveform Diffusion Models for Foley Sound Generationcited by 0
  8. 2026AI-Based Sound Effect Generation: A Narrative Review of Generative Models Across Input Modalitiescited by 0