# Swoosh! Rattle! Thump! -- Actions that Sound

arXiv: https://arxiv.org/abs/2007.01851 · HTML: https://mlsanity.com/paper/2007.01851/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2020.

## Built on by (most cited first, 15 of 15)

- [ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations](https://mlsanity.com/paper/2109.07991/) (2021)
- [See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2212.03858/) (2022)
- [MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation](https://mlsanity.com/paper/2310.16917/) (2023)
- [Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning](https://mlsanity.com/paper/2205.14850/) (2022)
- [Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos](https://mlsanity.com/paper/2406.09272/) (2024)
- [Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation](https://mlsanity.com/paper/2405.08576/) (2024)
- [That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation](https://mlsanity.com/paper/2210.01116/) (2022)
- [SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos](https://mlsanity.com/paper/2404.05206/) (2024)
- [All the Feels: A dexterous hand with large-area tactile sensing](https://mlsanity.com/paper/2210.15658/) (2022)
- [The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio](https://mlsanity.com/paper/2507.02864/) (2025)
- [Learning Sequential Latent Variable Models from Multimodal Time Series Data](https://mlsanity.com/paper/2204.10419/) (2022)
- [The Boombox: Visual Reconstruction from Acoustic Vibrations](https://mlsanity.com/paper/2105.08052/) (2021)
- [A Framework for Multisensory Foresight for Embodied Agents](https://mlsanity.com/paper/2109.07561/) (2021)
- [ANAVI: Audio Noise Awareness using Visuals of Indoor environments for NAVIgation](https://mlsanity.com/paper/2410.18932/) (2024)
- [Prosody as a Teaching Signal for Agent Learning: Exploratory Studies and Algorithmic Implications](https://mlsanity.com/paper/2410.23554/) (2024)

## Abstract (arXiv)

Truly intelligent agents need to capture the interplay of all their senses to build a rich physical understanding of their world. In robotics, we have seen tremendous progress in using visual and tactile perception; however, we have often ignored a key sense: sound. This is primarily due to the lack of data that captures the interplay of action and sound. In this work, we perform the first large-scale study of the interactions between sound and robotic action. To do this, we create the largest available sound-action-vision dataset with 15,000 interactions on 60 objects using our robotic platform Tilt-Bot. By tilting objects and allowing them to crash into the walls of a robotic tray, we collect rich four-channel audio information. Using this data, we explore the synergies between sound and action and present three key insights. First, sound is indicative of fine-grained object class information, e.g., sound can differentiate a metal screwdriver from a metal wrench. Second, sound also contains information about the causal effects of an action, i.e. given the sound produced, we can predict what action was applied to the object. Finally, object representations derived from audio embeddings are indicative of implicit physical properties. We demonstrate that on previously unseen objects, audio embeddings generated through interactions can predict forward models 24% better than passive visual embeddings. Project videos and data are at https://dhiraj100892.github.io/swoosh/