mlsanityData to 7 Oct 2026

Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning

arXiv 2205.14850 ↗First posted 30 May 2022cs.RO cs.LG cs.SDcited by 22

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Humans are capable of completing a range of challenging manipulation tasks that require reasoning jointly over modalities such as vision, touch, and sound. Moreover, many such tasks are partially-observed; for example, taking a notebook out of a backpack will lead to visual occlusion and require reasoning over the history of audio or tactile information. While robust tactile sensing can be costly to capture on robots, microphones near or on a robot's gripper are a cheap and easy way to acquire audio feedback of contact events, which can be a surprisingly valuable data source for perception in the absence of vision. Motivated by the potential for sound to mitigate visual occlusion, we aim to learn a set of challenging partially-observed manipulation tasks from visual and audio inputs. Our proposed system learns these tasks by combining offline imitation learning from a modest number of tele-operated demonstrations and online finetuning using human provided interventions. In a set of simulated tasks, we find that our system benefits from using audio, and that by using online interventions we are able to improve the success rate of offline imitation learning by ~20%. Finally, we find that our system can complete a set of challenging, partially-observed tasks on a Franka Emika Panda robot, like extracting keys from a bag, with a 70% success rate, 50% higher than a policy that does not use audio.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 60 of its 72 references that are in the map. First posted on arXiv 30 May 2022.

Builds on

60 of 72 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  2. 2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale9 Nov
  3. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learning31 Mar
  4. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperation12 Dec

Other references · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  6. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  11. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
38 more
  1. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  2. 2017Objects that Soundcited by 254
  3. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  4. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  5. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  6. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  7. 2016Learning to Navigate in Complex Environmentscited by 192
  8. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
  9. 2015Visually Indicated Soundscited by 128
  10. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  11. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  12. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 100
  13. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  14. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  15. 2018Visual Representations for Semantic Target Driven Navigationcited by 89
  16. 2017Neural Map: Structured Memory for Deep Reinforcement Learningcited by 88
  17. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  18. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 67
  19. 2019Connecting Touch and Vision via Cross-Modal Predictioncited by 66
  20. 2016Control of Memory, Active Perception, and Action in Minecraftcited by 61
  21. 2020See, Hear, Explore: Curiosity via Audio-Visual Associationcited by 43
  22. 20203D Shape Reconstruction from Vision and Touchcited by 42
  23. 2020Learning 3D Dynamic Scene Representations for Robot Manipulationcited by 39
  24. 2019Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Cluttercited by 30
  25. 2020Event-Driven Visual-Tactile Sensing and Learning for Robotscited by 27
  26. 2021Residual Reinforcement Learning from Demonstrationscited by 27
  27. 2020Robust, Occlusion-aware Pose Estimation for Objects Grasped by Adaptive Handscited by 26
  28. 2018Reinforcement Learning of Active Vision for Manipulating Objects under Occlusionscited by 25
  29. 2020Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learningcited by 23
  30. 2021VisuoSpatial Foresight for Physical Sequential Fabric Manipulationcited by 18
  31. 2021Structure from Silence: Learning Scene Structure from Ambient Soundcited by 17
  32. 2020Swoosh! Rattle! Thump! -- Actions that Soundcited by 15
  33. 2021Occlusion-robust Deformable Object Tracking without Physics Simulationcited by 15
  34. 2016Probabilistic Articulated Real-Time Tracking for Robot Manipulationcited by 13
  35. 2020Occlusion-Aware Search for Object Retrieval in Cluttercited by 11
  36. 2021Contact-Rich Manipulation of a Flexible Object based on Deep Predictive Learning using Vision and Tactilitycited by 9
  37. 2020Understanding Multi-Modal Perception Using Behavioral Cloning for Peg-In-a-Hole Insertion Taskscited by 8
  38. 2020X-Ray: Mechanical Search for an Occluded Object by Minimizing Support of Learned Occupancy Distributionscited by 7

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 3; 2024: 4; 2025: 6; 2026: 4.

22 papers.

Sort:

By area:

  1. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 61
  2. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 40
  3. 2023MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulationcited by 30
  4. 2024ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Datacited by 29
  5. 2024Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulationcited by 19
  6. 2022Transformers are Adaptable Task Plannerscited by 15
  7. 2024Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulationcited by 14
  8. 2022That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulationcited by 12
  9. 2025Robotic Compliant Object Prying Using Diffusion Policy Guided by Vision and Force Observationscited by 9
  10. 2022All the Feels: A dexterous hand with large-area tactile sensingcited by 7
  11. 2022Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrationscited by 6
  12. 2023Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Modelscited by 5
10 more
  1. 2025The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audiocited by 5
  2. 2025Multimodal Diffusion Forcing for Forceful Manipulationcited by 3
  3. 2025Force-Modulated Visual Policy for Robot-Assisted Dressing with Arm Motionscited by 2
  4. 2023CasIL: Cognizing and Imitating Skills via a Dual Cognition-Action Architecturecited by 0
  5. 2024ANAVI: Audio Noise Awareness using Visuals of Indoor environments for NAVIgationcited by 0
  6. 2025TranTac: Leveraging Transient Tactile Signals for Contact-Rich Robotic Manipulationcited by 0
  7. 2026Hierarchical Audio-Visual-Proprioceptive Fusion for Precise Robotic Manipulationcited by 0
  8. 2026S2A2: Audio-Visual Imitation Learning for Manipulation Tasks Using Acoustic Spatial Informationcited by 0
  9. 2026Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generationcited by 0
  10. 2026Learning tactile perception from high-bandwidth single-point sensingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MicCheck: Repurposing Off-the-Shelf Pin Microphones for Easy and Low-Cost Contact Sensing ↗
  2. 2026You're Pushing My Buttons: Instrumented Learning of Gentle Button Presses ↗
  3. 2024VITaL Pretraining: Visuo-Tactile Pretraining for Tactile and Non-Tactile Manipulation Policies
  4. 2026Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation
  5. 2020Visual Imitation Made Easy
  6. 2025Learning Robot Manipulation from Audio World Models
  7. 2023Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor
  8. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play