mlsanityData to 14 Oct 2026

S2A2: Audio-Visual Imitation Learning for Manipulation Tasks Using Acoustic Spatial Information

arXiv 2607.26047 ↗First posted 28 Jul 2026cs.ROcited by 0

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Acoustic information provides rich cues about object location, material properties, and changes caused by contact or motion. This paper introduces a new set of acoustic-aware manipulation tasks for imitation learning, in which robots must use auditory cues to determine manipulation targets. These tasks require sound source localization and identification for active exploration in robotic manipulation. Also, we propose a multimodal imitation learning framework, Spatial-Spectral Audio Action (S2A2), that integrates visual features with acoustic spatial and acoustic signal information for the acoustic-aware manipulation tasks. We implemented S2A2 models that integrates policies such as ACT, Diffusion Policy, VQ-BeT, and π0, into our framework. Simulation experiments showed that the proposed method is the most effective for tasks requiring both position and timbre. Furthermore, real-robot experiments confirm the applicability of the proposed tasks and framework to real-world manipulation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 41 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

25 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor2 Nov
  2. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  3. 2022Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning30 May

Other references · 22

  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  2. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  3. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  4. 2024Behavior Generation with Latent Actionscited by 160
  5. 2017Pyroomacoustics: A Python package for audio room simulations and array processing algorithmscited by 132
  6. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 68
  7. 2024ALOHA 2: An Enhanced Low-Cost Hardware for Bimanual Teleoperationcited by 63
  8. 2022Visuo-Tactile Transformers for Manipulationcited by 47
  9. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  10. 2024ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Datacited by 31
  11. 2025ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interfacecited by 23
  12. 2024Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulationcited by 14
10 more
  1. 2022That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulationcited by 12
  2. 2019Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouringcited by 11
  3. 2025The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audiocited by 6
  4. 2025Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulationcited by 6
  5. 2023MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Learning via Interactive Perceptioncited by 5
  6. 2024SLAM-based Joint Calibration of Multiple Asynchronous Microphone Arrays and Sound Source Localizationcited by 4
  7. 2025CAVER: Curious Audiovisual Exploring Robotcited by 2
  8. 2025Audio-VLA: Adding Contact Audio Perception to Vision-Language-Action Model for Robotic Manipulationcited by 2
  9. 2018Deep Neural Object Analysis by Interactive Auditory Exploration with a Humanoid Robotcited by 1
  10. 2025A Review on Sound Source Localization in Robotics: Focusing on Deep Learning Methodscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Listen Before Act: Audio-Conditioned Robot Manipulation with ListenVLA
  2. 2025Learning Robot Manipulation from Audio World Models
  3. 2025Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models
  4. 2025SAM2Grasp: Resolve Multi-modal Grasping via Prompt-conditioned Temporal Action Prediction
  5. 2025MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulation
  6. 2025Object-Centric Mobile Manipulation through SAM2-Guided Perception and Imitation Learning
  7. 2026Dynamics-Aware Meta-Imitation for Generalization to Unseen Robotic Manipulation
  8. 2025RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environments