mlsanityData to 7 Oct 2026

ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data

arXiv 2406.19464 ↗First posted 27 Jun 2024cs.RO cs.AI cs.CVcited by 29

Not on the map. Why?

Abstract (arXiv)

Audio signals provide rich information for the robot interaction and object properties through contact. This information can surprisingly ease the learning of contact-rich robot manipulation skills, especially when the visual information alone is ambiguous or incomplete. However, the usage of audio data in robot manipulation has been constrained to teleoperated demonstrations collected by either attaching a microphone to the robot or object, which significantly limits its usage in robot learning pipelines. In this work, we introduce ManiWAV: an 'ear-in-hand' data collection device to collect in-the-wild human demonstrations with synchronous audio and visual feedback, and a corresponding policy interface to learn robot manipulation policy directly from the demonstrations. We demonstrate the capabilities of our system through four contact-rich manipulation tasks that require either passively sensing the contact events and modes, or actively sensing the object surface materials and states. In addition, we show that our system can generalize to unseen in-the-wild environments by learning from diverse in-the-wild human demonstrations.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

31 of 46 references.

References · 31

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  3. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  4. 2021AST: Audio Spectrogram Transformercited by 634
  5. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  6. 2018An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solutioncited by 302
  7. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  8. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  9. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
  10. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 73
  11. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 61
  12. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 60
19 more
  1. 2022Visuo-Tactile Transformers for Manipulationcited by 43
  2. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 40
  3. 2022Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Featurescited by 37
  4. 2022Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learningcited by 22
  5. 2020Soft-Bubble grippers for robust and perceptive manipulationcited by 18
  6. 2019Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulationcited by 15
  7. 2024Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulationcited by 14
  8. 2022VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objectscited by 13
  9. 2022TANDEM: Learning Joint Exploration and Decision Making with Tactile Sensorscited by 12
  10. 2022That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulationcited by 12
  11. 2022TANDEM3D: Active Tactile Exploration for 3D Object Recognitioncited by 10
  12. 2022DelTact: A Vision-based Tactile Sensor Using Dense Color Patterncited by 7
  13. 2024CushSense: Soft, Stretchable, and Comfortable Tactile-Sensing Skin for Physical Human-Robot Interactioncited by 7
  14. 2022Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensingcited by 5
  15. 2024MResT: Multi-Resolution Sensing for Real-Time Control with Vision-Language Modelscited by 5
  16. 2021AuraSense: Robot Collision Avoidance by Full Surface Proximity Detectioncited by 4
  17. 2024Tactile-based Object Retrieval From Granular Mediacited by 4
  18. 2024Sensorized Soft Skin for Dexterous Robotic Handscited by 4
  19. 2023Active Acoustic Sensing for Robot Manipulationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 17; 2026: 11.

29 papers.

Sort:

By area:

  1. 2026Causal World Modeling for Robot Controlcited by 289
  2. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  3. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 40
  4. 2025Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Grippercited by 29
  5. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 17
  6. 2025PolyTouch: A Robust Multi-Modal Tactile Sensor for Contact-rich Manipulation Using Tactile-Diffusion Policiescited by 13
  7. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 12
  8. 2025Robotic Compliant Object Prying Using Diffusion Policy Guided by Vision and Force Observationscited by 9
  9. 2025Policy Contrastive Decoding for Robotic Foundation Modelscited by 8
  10. 2025The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audiocited by 5
  11. 2025Kaiwu: A Multimodal Manipulation Dataset and Framework for Robot Learning and Human-Robot Interactioncited by 4
  12. 2025On the Importance of Tactile Sensing for Imitation Learning: A Case Study on Robotic Match Lightingcited by 4
17 more
  1. 2025SoundReactor: Frame-level Online Video-to-Audio Generationcited by 4
  2. 2025TouchFormer: A Robust Transformer-based Framework for Multimodal Material Perceptioncited by 4
  3. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  4. 2024Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videoscited by 3
  5. 2025Multimodal Diffusion Forcing for Forceful Manipulationcited by 3
  6. 2025Learning Gentle Grasping Using Vision, Sound, and Touchcited by 2
  7. 2025Learning Robot Manipulation from Audio World Modelscited by 2
  8. 2026YUBI: Yielding Universal Bidigital Interface for Bimanual Dexterous Manipulation at Scalecited by 2
  9. 2025Audio-VLA: Adding Contact Audio Perception to Vision-Language-Action Model for Robotic Manipulationcited by 1
  10. 2026Selective Perception for Robot: Task-Aware Attention in Multimodal VLAcited by 1
  11. 2026Spatially Prompted Visual Trajectory Prediction for Egocentric Manipulationcited by 1
  12. 2025TranTac: Leveraging Transient Tactile Signals for Contact-Rich Robotic Manipulationcited by 0
  13. 2026Hierarchical Audio-Visual-Proprioceptive Fusion for Precise Robotic Manipulationcited by 0
  14. 2026Asynchronous Multimodal Diffusion Policy Composition via Latency-Aware Guidance Fusioncited by 0
  15. 2026S2A2: Audio-Visual Imitation Learning for Manipulation Tasks Using Acoustic Spatial Informationcited by 0
  16. 2026Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generationcited by 0
  17. 2026Learning tactile perception from high-bandwidth single-point sensingcited by 0