# ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data

arXiv: https://arxiv.org/abs/2406.19464 · HTML: https://mlsanity.com/paper/2406.19464/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 30 of 31)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction](https://mlsanity.com/paper/2306.15724/) (2023)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer](https://mlsanity.com/paper/2204.02389/) (2022)
- [See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2212.03858/) (2022)
- [Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play](https://mlsanity.com/paper/2303.12076/) (2023)
- [Visuo-Tactile Transformers for Manipulation](https://mlsanity.com/paper/2210.00121/) (2022)
- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features](https://mlsanity.com/paper/2209.13042/) (2022)
- [Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning](https://mlsanity.com/paper/2205.14850/) (2022)
- [Soft-Bubble grippers for robust and perceptive manipulation](https://mlsanity.com/paper/2004.03691/) (2020)
- [Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation](https://mlsanity.com/paper/1904.02252/) (2019)
- [Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation](https://mlsanity.com/paper/2405.08576/) (2024)
- [VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects](https://mlsanity.com/paper/2210.03701/) (2022)
- [TANDEM: Learning Joint Exploration and Decision Making with Tactile Sensors](https://mlsanity.com/paper/2203.00798/) (2022)
- [That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation](https://mlsanity.com/paper/2210.01116/) (2022)
- [TANDEM3D: Active Tactile Exploration for 3D Object Recognition](https://mlsanity.com/paper/2209.08772/) (2022)
- [CushSense: Soft, Stretchable, and Comfortable Tactile-Sensing Skin for Physical Human-Robot Interaction](https://mlsanity.com/paper/2405.03155/) (2024)
- [DelTact: A Vision-based Tactile Sensor Using Dense Color Pattern](https://mlsanity.com/paper/2202.02179/) (2022)
- [Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing](https://mlsanity.com/paper/2210.04026/) (2022)
- [MResT: Multi-Resolution Sensing for Real-Time Control with Vision-Language Models](https://mlsanity.com/paper/2401.14502/) (2024)
- [Tactile-based Object Retrieval From Granular Media](https://mlsanity.com/paper/2402.04536/) (2024)
- [AuraSense: Robot Collision Avoidance by Full Surface Proximity Detection](https://mlsanity.com/paper/2108.04867/) (2021)
- [Sensorized Soft Skin for Dexterous Robotic Hands](https://mlsanity.com/paper/2404.19448/) (2024)

## Built on by (most cited first, 30 of 31)

- [Causal World Modeling for Robot Control](https://mlsanity.com/paper/2601.21998/) (2026)
- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding](https://mlsanity.com/paper/2501.04693/) (2025)
- [Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper](https://mlsanity.com/paper/2507.15062/) (2025)
- [AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletons](https://mlsanity.com/paper/2503.03081/) (2025)
- [PolyTouch: A Robust Multi-Modal Tactile Sensor for Contact-rich Manipulation Using Tactile-Diffusion Policies](https://mlsanity.com/paper/2504.19341/) (2025)
- [Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directions](https://mlsanity.com/paper/2502.15336/) (2025)
- [Robotic Compliant Object Prying Using Diffusion Policy Guided by Vision and Force Observations](https://mlsanity.com/paper/2503.03998/) (2025)
- [Policy Contrastive Decoding for Robotic Foundation Models](https://mlsanity.com/paper/2505.13255/) (2025)
- [The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio](https://mlsanity.com/paper/2507.02864/) (2025)
- [Data Pyramid for Embodied Manipulation: A Survey](https://mlsanity.com/paper/2607.24744/) (2026)
- [Kaiwu: A Multimodal Manipulation Dataset and Framework for Robot Learning and Human-Robot Interaction](https://mlsanity.com/paper/2503.05231/) (2025)
- [SoundReactor: Frame-level Online Video-to-Audio Generation](https://mlsanity.com/paper/2510.02110/) (2025)
- [On the Importance of Tactile Sensing for Imitation Learning: A Case Study on Robotic Match Lighting](https://mlsanity.com/paper/2504.13618/) (2025)
- [Multimodal Diffusion Forcing for Forceful Manipulation](https://mlsanity.com/paper/2511.04812/) (2025)
- [TouchFormer: A Robust Transformer-based Framework for Multimodal Material Perception](https://mlsanity.com/paper/2511.19509/) (2025)
- [YUBI: Yielding Universal Bidigital Interface for Bimanual Dexterous Manipulation at Scale](https://mlsanity.com/paper/2606.10244/) (2026)
- [Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos](https://mlsanity.com/paper/2410.09286/) (2024)
- [Learning Robot Manipulation from Audio World Models](https://mlsanity.com/paper/2512.08405/) (2025)
- [Learning Gentle Grasping Using Vision, Sound, and Touch](https://mlsanity.com/paper/2503.07926/) (2025)
- [Audio-VLA: Adding Contact Audio Perception to Vision-Language-Action Model for Robotic Manipulation](https://mlsanity.com/paper/2511.09958/) (2025)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [Selective Perception for Robot: Task-Aware Attention in Multimodal VLA](https://mlsanity.com/paper/2602.15543/) (2026)
- [Spatially Prompted Visual Trajectory Prediction for Egocentric Manipulation](https://mlsanity.com/paper/2605.20085/) (2026)
- [Asynchronous Multimodal Diffusion Policy Composition via Latency-Aware Guidance Fusion](https://mlsanity.com/paper/2607.17257/) (2026)
- [TranTac: Leveraging Transient Tactile Signals for Contact-Rich Robotic Manipulation](https://mlsanity.com/paper/2509.16550/) (2025)
- [Hierarchical Audio-Visual-Proprioceptive Fusion for Precise Robotic Manipulation](https://mlsanity.com/paper/2602.13640/) (2026)
- [S2A2: Audio-Visual Imitation Learning for Manipulation Tasks Using Acoustic Spatial Information](https://mlsanity.com/paper/2607.26047/) (2026)
- [Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generation](https://mlsanity.com/paper/2609.19137/) (2026)
- [Learning tactile perception from high-bandwidth single-point sensing](https://mlsanity.com/paper/2609.24621/) (2026)

## Abstract (arXiv)

Audio signals provide rich information for the robot interaction and object properties through contact. This information can surprisingly ease the learning of contact-rich robot manipulation skills, especially when the visual information alone is ambiguous or incomplete. However, the usage of audio data in robot manipulation has been constrained to teleoperated demonstrations collected by either attaching a microphone to the robot or object, which significantly limits its usage in robot learning pipelines. In this work, we introduce ManiWAV: an 'ear-in-hand' data collection device to collect in-the-wild human demonstrations with synchronous audio and visual feedback, and a corresponding policy interface to learn robot manipulation policy directly from the demonstrations. We demonstrate the capabilities of our system through four contact-rich manipulation tasks that require either passively sensing the contact events and modes, or actively sensing the object surface materials and states. In addition, we show that our system can generalize to unseen in-the-wild environments by learning from diverse in-the-wild human demonstrations.