# Multisensory Learning Framework for Robot Drumming

arXiv: https://arxiv.org/abs/1907.09775 · HTML: https://mlsanity.com/paper/1907.09775/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2019.

## Built on by (most cited first, 1 of 1)

- [Robot Grasping and Manipulation: A Prospective](https://mlsanity.com/paper/2303.07807/) (2023)

## Abstract (arXiv)

The hype about sensorimotor learning is currently reaching high fever, thanks to the latest advancement in deep learning. In this paper, we present an open-source framework for collecting large-scale, time-synchronised synthetic data from highly disparate sensory modalities, such as audio, video, and proprioception, for learning robot manipulation tasks. We demonstrate the learning of non-linear sensorimotor mappings for a humanoid drumming robot that generates novel motion sequences from desired audio data using cross-modal correspondences. We evaluate our system through the quality of its cross-modal retrieval, for generating suitable motion sequences to match desired unseen audio or video sequences.