mlsanityData to 7 Oct 2026

Visual to Sound: Generating Natural Sound for Videos in the Wild

arXiv 1712.01393 ↗First posted 4 Dec 2017cs.CVcited by 98

Not on the map. Why?

Abstract (arXiv)

As two of the five traditional human senses (sight, hearing, taste, smell, and touch), vision and sound are basic sources through which humans understand the world. Often correlated during natural events, these two modalities combine to jointly affect human perception. In this paper, we pose the task of generating sound given visual input. Such capabilities could help enable applications in virtual reality (generating sound for virtual scenes automatically) or provide additional accessibility to images or videos for people with visual impairments. As a first step in this direction, we apply learning-based methods to generate raw waveform samples given input video frames. We evaluate our models on a dataset of videos containing a variety of sounds (such as ambient sounds and sounds from people/animals). Our experiments show that the generated sounds are fairly realistic and have good temporal synchronization with the visual inputs.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 16; 2023: 22; 2024: 18; 2025: 13; 2026: 13.

98 papers.

Sort:

By area:

  1. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  2. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 225
  3. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  4. 2021Taming Visually Guided Sound Generationcited by 136
  5. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 133
  6. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  7. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  8. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  9. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  10. 2021Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advancescited by 54
  11. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  12. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
38 more
  1. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  2. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
  3. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 40
  4. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 38
  5. 2023Connecting Multi-modal Contrastive Representationscited by 36
  6. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  7. 2021Visual Scene Graphs for Audio Source Separationcited by 29
  8. 2022A Review on Methods and Applications in Multimodal Deep Learningcited by 29
  9. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  10. 2024Text-to-Audio Generation Synchronized with Videoscited by 28
  11. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  12. 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
  13. 2021Can audio-visual integration strengthen robustness under multimodal attacks?cited by 23
  14. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
  15. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  16. 2022Learning Visual Styles from Audio-Visual Associationscited by 22
  17. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  18. 2023Physics-Driven Diffusion Models for Impact Sound Synthesis from Videoscited by 21
  19. 2021FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videoscited by 20
  20. 2023V2Meow: Meowing to the Visual Beat via Video-to-Music Generationcited by 18
  21. 2023XVO: Generalized Visual Odometry via Cross-Modal Self-Trainingcited by 17
  22. 2024VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generationcited by 16
  23. 2021Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generationcited by 14
  24. 2024VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videoscited by 14
  25. 2023Preserving Modality Structure Improves Multi-Modal Learningcited by 13
  26. 2022Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separationcited by 11
  27. 2022VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglementcited by 11
  28. 2025ViSAGe: Video-to-Spatial Audio Generationcited by 11
  29. 2022KazakhTTS2: Extending the Open-Source Kazakh TTS Corpus With More Data, Speakers, and Topicscited by 10
  30. 2024SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Soundcited by 10
  31. 2022A Proposal for Foley Sound Synthesis Challengecited by 8
  32. 2023LAVSS: Location-Guided Audio-Visual Spatial Audio Separationcited by 8
  33. 2024Self-Supervised Audio-Visual Soundscape Stylizationcited by 8
  34. 2024Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Compositioncited by 8
  35. 2025Long-Video Audio Synthesis with Multi-Agent Collaborationcited by 8
  36. 2025VGGSounder: Audio-Visual Evaluations for Foundation Modelscited by 8
  37. 2024Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesiscited by 7
  38. 2021Collaborative Learning to Generate Audio-Video Jointlycited by 6

50 most cited of 98 All 98 (JSON)