# Learning Multimodal Contact-Rich Skills from Demonstrations Without Reward Engineering

arXiv: https://arxiv.org/abs/2103.01296 · HTML: https://mlsanity.com/paper/2103.01296/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2021.

## Built on by (most cited first, 2 of 2)

- [Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning](https://mlsanity.com/paper/2205.14850/) (2022)
- [Learning from Demonstration Framework for Multi-Robot Systems Using Interaction Keypoints and Soft Actor-Critic Methods](https://mlsanity.com/paper/2404.02324/) (2024)

## Abstract (arXiv)

Everyday contact-rich tasks, such as peeling, cleaning, and writing, demand multimodal perception for effective and precise task execution. However, these present a novel challenge to robots as they lack the ability to combine these multimodal stimuli for performing contact-rich tasks. Learning-based methods have attempted to model multi-modal contact-rich tasks, but they often require extensive training examples and task-specific reward functions which limits their practicality and scope. Hence, we propose a generalizable model-free learning-from-demonstration framework for robots to learn contact-rich skills without explicit reward engineering. We present a novel multi-modal sensor data representation which improves the learning performance for contact-rich skills. We performed training and experiments using the real-life Sawyer robot for three everyday contact-rich skills -- cleaning, writing, and peeling. Notably, the framework achieves a success rate of 100% for the peeling and writing skill, and 80% for the cleaning skill. Hence, this skill learning framework can be extended for learning other physical manipulation skills.