# An End-to-End Human Simulator for Task-Oriented Multimodal Human-Robot Collaboration

arXiv: https://arxiv.org/abs/2304.00584 · HTML: https://mlsanity.com/paper/2304.00584/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2023.

## Built on by (most cited first, 1 of 1)

- [Learning to Plan in Human-Robot Collaboration: Multimodal Reinforcement Learning for Adaptive Interaction](https://mlsanity.com/paper/2609.25274/) (2026)

## Abstract (arXiv)

This paper proposes a neural network-based user simulator that can provide a multimodal interactive environment for training Reinforcement Learning (RL) agents in collaborative tasks involving multiple modes of communication. The simulator is trained on the existing ELDERLY-AT-HOME corpus and accommodates multiple modalities such as language, pointing gestures, and haptic-ostensive actions. The paper also presents a novel multimodal data augmentation approach, which addresses the challenge of using a limited dataset due to the expensive and time-consuming nature of collecting human demonstrations. Overall, the study highlights the potential for using RL and multimodal user simulators in developing and improving domestic assistive robots.