# Visually Indicated Sounds

arXiv: https://arxiv.org/abs/1512.08512 · HTML: https://mlsanity.com/paper/1512.08512/

## Key facts

- Cited by 128 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2015.

## Built on by (most cited first, 30 of 128)

- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2306.17203/) (2023)
- [MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis](https://mlsanity.com/paper/2412.15322/) (2024)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds](https://mlsanity.com/paper/2407.01494/) (2024)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models](https://mlsanity.com/paper/2308.09300/) (2023)
- [Mix and Localize: Localizing Sound Sources in Mixtures](https://mlsanity.com/paper/2211.15058/) (2022)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Conditional Generation of Audio from Video via Foley Analogies](https://mlsanity.com/paper/2304.08490/) (2023)
- [ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning](https://mlsanity.com/paper/2101.10803/) (2021)
- [Video-Guided Foley Sound Generation with Multimodal Controls](https://mlsanity.com/paper/2411.17698/) (2024)
- [Read, Watch and Scream! Sound Generation from Text and Video](https://mlsanity.com/paper/2407.05551/) (2024)
- [AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?](https://mlsanity.com/paper/2412.02611/) (2024)
- [Visually Informed Binaural Audio Generation without Binaural Audios](https://mlsanity.com/paper/2104.06162/) (2021)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation](https://mlsanity.com/paper/2210.12649/) (2022)
- [Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment](https://mlsanity.com/paper/2303.17490/) (2023)
- [MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World](https://mlsanity.com/paper/2401.08577/) (2024)
- [Tell What You Hear From What You See -- Video to Audio Generation Through Text](https://mlsanity.com/paper/2411.05679/) (2024)
- [Audio-Synchronized Visual Animation](https://mlsanity.com/paper/2403.05659/) (2024)
- [Self-supervised object detection from audio-visual correspondence](https://mlsanity.com/paper/2104.06401/) (2021)
- [Visual Scene Graphs for Audio Source Separation](https://mlsanity.com/paper/2109.11955/) (2021)
- [iQuery: Instruments as Queries for Audio-Visual Sound Separation](https://mlsanity.com/paper/2212.03814/) (2022)
- [Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity](https://mlsanity.com/paper/2407.10387/) (2024)
- [End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks](https://mlsanity.com/paper/2104.13332/) (2021)
- [Tactile-Augmented Radiance Fields](https://mlsanity.com/paper/2405.04534/) (2024)
- [Self-supervised Audiovisual Representation Learning for Remote Sensing Data](https://mlsanity.com/paper/2108.00688/) (2021)
- [MESH2IR: Neural Acoustic Impulse Response Generator for Complex 3D Scenes](https://mlsanity.com/paper/2205.09248/) (2022)

## Abstract (arXiv)

Objects make distinctive sounds when they are hit or scratched. These sounds reveal aspects of an object's material properties, as well as the actions that produced them. In this paper, we propose the task of predicting what sound an object makes when struck as a way of studying physical interactions within a visual scene. We present an algorithm that synthesizes sound from silent videos of people hitting and scratching objects with a drumstick. This algorithm uses a recurrent neural network to predict sound features from videos and then produces a waveform from these features with an example-based synthesis procedure. We show that the sounds predicted by our model are realistic enough to fool participants in a "real or fake" psychophysical experiment, and that they convey significant information about material properties and physical interactions.