mlsanityData to 14 Oct 2026

Visually Indicated Sounds

arXiv 1512.08512 ↗First posted 28 Dec 2015cs.CV cs.LG cs.SDcited by 128

Not on the map. Why?

Abstract (arXiv)

Objects make distinctive sounds when they are hit or scratched. These sounds reveal aspects of an object's material properties, as well as the actions that produced them. In this paper, we propose the task of predicting what sound an object makes when struck as a way of studying physical interactions within a visual scene. We present an algorithm that synthesizes sound from silent videos of people hitting and scratching objects with a drumstick. This algorithm uses a recurrent neural network to predict sound features from videos and then produces a waveform from these features with an example-based synthesis procedure. We show that the sounds predicted by our model are realistic enough to fool participants in a "real or fake" psychophysical experiment, and that they convey significant information about material properties and physical interactions.

Key facts. Cited by 128 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Dec 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 23; 2022: 19; 2023: 20; 2024: 28; 2025: 23; 2026: 15.

128 papers.

Sort:

By area:

  1. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  2. 2021Taming Visually Guided Sound Generationcited by 136
  3. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 134
  4. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  5. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  6. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
  7. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 87
  8. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  9. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  10. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  11. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  12. 2021ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learningcited by 50
38 more
  1. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  2. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  3. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 47
  4. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  5. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  6. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  7. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
  8. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 39
  9. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  10. 2024Audio-Synchronized Visual Animationcited by 32
  11. 2021Self-supervised object detection from audio-visual correspondencecited by 29
  12. 2021Visual Scene Graphs for Audio Source Separationcited by 29
  13. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  14. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  15. 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
  16. 2024Tactile-Augmented Radiance Fieldscited by 26
  17. 2021Self-supervised Audiovisual Representation Learning for Remote Sensing Datacited by 24
  18. 2022MESH2IR: Neural Acoustic Impulse Response Generator for Complex 3D Scenescited by 23
  19. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
  20. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  21. 2022Learning Visual Styles from Audio-Visual Associationscited by 22
  22. 2022Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learningcited by 22
  23. 2023Physics-Driven Diffusion Models for Impact Sound Synthesis from Videoscited by 22
  24. 2024Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videoscited by 21
  25. 2021FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videoscited by 20
  26. 2023CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Modelscited by 20
  27. 2024A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generationcited by 19
  28. 2022It's Time for Artistic Correspondence in Music and Videocited by 18
  29. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  30. 2023V2Meow: Meowing to the Visual Beat via Video-to-Music Generationcited by 18
  31. 2021Structure from Silence: Learning Scene Structure from Ambient Soundcited by 17
  32. 2023Unsupervised Object Localization in the Era of Self-Supervised ViTs: A Surveycited by 16
  33. 2024Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Soundcited by 16
  34. 2023Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generationcited by 15
  35. 2023AdVerb: Visually Guided Audio Dereverberationcited by 15
  36. 2024SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Modelscited by 15
  37. 2024VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videoscited by 15
  38. 2021Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generationcited by 14

50 most cited of 128 All 128 (JSON)