mlsanityData to 7 Oct 2026

Conditional Generation of Audio from Video via Foley Analogies

arXiv 2304.08490 ↗First posted 17 Apr 2023cs.CV cs.MM cs.SDcited by 52

Not on the map. Why?

Abstract (arXiv)

The sound effects that designers add to videos are designed to convey a particular artistic effect and, thus, may be quite different from a scene's true sound. Inspired by the challenges of creating a soundtrack for a video that differs from its true sound, but that nonetheless matches the actions occurring on screen, we propose the problem of conditional Foley. We present the following contributions to address this problem. First, we propose a pretext task for training our model to predict sound for an input video clip using a conditional audio-visual clip sampled from another time within the same source video. Second, we propose a model for generating a soundtrack for a silent input video, given a user-supplied example that specifies what the video should "sound like". We show through human studies and automated evaluation metrics that our model successfully generates sound from video, while varying its output according to the content of a supplied example. Project site: https://xypb.github.io/CondFoleyGen/

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 44 of its 76 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

44 of 76 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  8. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  9. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  10. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  11. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  12. 2015A Neural Algorithm of Artistic Stylecited by 480
32 more
  1. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  2. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  3. 2018The Sound of Pixelscited by 258
  4. 2017Objects that Soundcited by 254
  5. 2021Taming Visually Guided Sound Generationcited by 136
  6. 2015Visually Indicated Soundscited by 128
  7. 2017Real-Time User-Guided Image Colorization with Learned Deep Priorscited by 123
  8. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  9. 2020Talking-head Generation with Rhythmic Head Motioncited by 112
  10. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  11. 2020Foley Music: Learning to Generate Music from Videoscited by 82
  12. 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
  13. 2020Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Soundscited by 55
  14. 2017Vid2speech: Speech Reconstruction from Silent Videocited by 53
  15. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  16. 2022Visual Acoustic Matchingcited by 45
  17. 2020Telling Left from Right: Learning Spatial Correspondence of Sight and Soundcited by 43
  18. 2021Sound-Guided Semantic Image Manipulationcited by 35
  19. 2020DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networkscited by 34
  20. 2018TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfercited by 32
  21. 2022Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectorscited by 25
  22. 2019Who said that?: Audio-visual speaker diarisation of real-world meetingscited by 22
  23. 2020Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movementscited by 22
  24. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  25. 2022Learning Visual Styles from Audio-Visual Associationscited by 22
  26. 2021FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videoscited by 20
  27. 2020AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos with Deep Learningcited by 19
  28. 2018Neural Style Transfer for Audio Spectogramscited by 18
  29. 2022It's Time for Artistic Correspondence in Music and Videocited by 18
  30. 2021Structure from Silence: Learning Scene Structure from Ambient Soundcited by 17
  31. 2021Who calls the shots? Rethinking Few-Shot Learning for Audiocited by 11
  32. 2022VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglementcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 24; 2025: 19; 2026: 7.

52 papers.

Sort:

By area:

  1. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  2. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  3. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  4. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  5. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  6. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 52
  7. 2024Temporally Aligned Audio for Video with Autoregressioncited by 51
  8. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  9. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  10. 2024TAVGBench: Benchmarking Text to Audible-Video Generationcited by 33
  11. 2024Text-to-Audio Generation Synchronized with Videoscited by 28
  12. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
38 more
  1. 2024Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videoscited by 21
  2. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 21
  3. 2025UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generationcited by 21
  4. 2024Language-Guided Joint Audio-Visual Editing via One-Shot Adaptationcited by 18
  5. 2024T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesiscited by 17
  6. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  7. 2024A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generationcited by 17
  8. 2024Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Soundcited by 16
  9. 2024VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videoscited by 14
  10. 2025Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decompositioncited by 10
  11. 2024Self-Supervised Audio-Visual Soundscape Stylizationcited by 8
  12. 2024Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Compositioncited by 8
  13. 2025Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformerscited by 7
  14. 2024The Sound of Water: Inferring Physical Properties from Pouring Liquidscited by 5
  15. 2024FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignmentcited by 5
  16. 2025TA-V2A: Textually Assisted Video-to-Audio Generationcited by 5
  17. 2025DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generationcited by 5
  18. 2025AudioMoG: Guiding Audio Generation with Mixture-of-Guidancecited by 5
  19. 2025Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillationcited by 4
  20. 2025SoundReactor: Frame-level Online Video-to-Audio Generationcited by 4
  21. 2024SonicVisionLM: Playing Sound with Vision Language Modelscited by 3
  22. 2025Extending Visual Dynamics for Video-to-Music Generationcited by 3
  23. 2025Sounding that Object: Interactive Object-Aware Image to Audio Generationcited by 3
  24. 2025Hear What Matters! Text-conditioned Selective Video-to-Audio Generationcited by 3
  25. 2024Smooth-Foley: Creating Continuous Sound for Video-to-Audio Generation Under Semantic Guidancecited by 2
  26. 2025Audio Texture Manipulation by Exemplar-Based Analogycited by 2
  27. 2025Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenescited by 2
  28. 2026Guided by the Plan: Enhancing Faithful Autoregressive Text-to-Audio Generation with Guided Decodingcited by 2
  29. 2026V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generationcited by 2
  30. 2026AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfercited by 2
  31. 2026FoleyDesigner: Immersive Stereo Foley Generation with Precise Spatio-Temporal Alignment for Film Clipscited by 1
  32. 2025EXPOTION: Facial Expression and Motion Control for Multimodal Music Generationcited by 0
  33. 2025Audio-Guided Visual Editing with Complex Multi-Modal Promptscited by 0
  34. 2025Learning from Silence and Noise for Visual Sound Source Localizationcited by 0
  35. 2025Efficient Video-to-Audio Generation via Multiple Foundation Models Mappercited by 0
  36. 2025StereoSync: Spatially-Aware Stereo Audio Generation from Videocited by 0
  37. 2025FoleyGRAM: Video-to-Audio Generation with GRAM-Aligned Multimodal Encoderscited by 0
  38. 2026MMAudio-LABEL: Audio Event Labeling via Audio Generation for Silent Videocited by 0

50 most cited of 52 All 52 (JSON)