# Learning from Demonstration Framework for Multi-Robot Systems Using Interaction Keypoints and Soft Actor-Critic Methods

arXiv: https://arxiv.org/abs/2404.02324 · HTML: https://mlsanity.com/paper/2404.02324/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 2 Apr 2024.

## Placed in

- Offline RL and imitation learning from data / [Robot learning from demonstration](https://mlsanity.com/offlinerl/robot-learning-from-demonstration/)

## Builds on (most cited first, 12 of 12)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [End-to-End Robotic Reinforcement Learning without Reward Engineering](https://mlsanity.com/paper/1904.07854/) (2019)
- [Interactive Policy Learning through Confidence-Based Autonomy](https://mlsanity.com/paper/1401.3439/) (2014)
- [A Vision-Guided Multi-Robot Cooperation Framework for Learning-by-Demonstration and Task Reproduction](https://mlsanity.com/paper/1706.00508/) (2017)
- [Extending Policy from One-Shot Learning through Coaching](https://mlsanity.com/paper/1905.04841/) (2019)
- [Investigating the Effect of Deictic Movements of a Multi-robot](https://mlsanity.com/paper/2006.03805/) (2020)
- [Learning Multimodal Contact-Rich Skills from Demonstrations Without Reward Engineering](https://mlsanity.com/paper/2103.01296/) (2021)
- [A Dataset Schema for Cooperative Learning from Demonstration in Multi-robots Systems](https://mlsanity.com/paper/1912.01741/) (2019)
- [UPPLIED: UAV Path Planning for Inspection through Demonstration](https://mlsanity.com/paper/2303.04284/) (2023)

## Abstract (arXiv)

Learning from Demonstration (LfD) is a promising approach to enable Multi-Robot Systems (MRS) to acquire complex skills and behaviors. However, the intricate interactions and coordination challenges in MRS pose significant hurdles for effective LfD. In this paper, we present a novel LfD framework specifically designed for MRS, which leverages visual demonstrations to capture and learn from robot-robot and robot-object interactions. Our framework introduces the concept of Interaction Keypoints (IKs) to transform the visual demonstrations into a representation that facilitates the inference of various skills necessary for the task. The robots then execute the task using sensorimotor actions and reinforcement learning (RL) policies when required. A key feature of our approach is the ability to handle unseen contact-based skills that emerge during the demonstration. In such cases, RL is employed to learn the skill using a classifier-based reward function, eliminating the need for manual reward engineering and ensuring adaptability to environmental changes. We evaluate our framework across a range of mobile robot tasks, covering both behavior-based and contact-based domains. The results demonstrate the effectiveness of our approach in enabling robots to learn complex multi-robot tasks and behaviors from visual demonstrations.