# HARMONIC: A Multimodal Dataset of Assistive Human-Robot Collaboration

arXiv: https://arxiv.org/abs/1807.11154 · HTML: https://mlsanity.com/paper/1807.11154/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jul 2018.

## Built on by (most cited first, 10 of 10)

- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2106.13948/) (2021)
- [LILA: Language-Informed Latent Actions](https://mlsanity.com/paper/2111.03205/) (2021)
- [Towards Modeling and Influencing the Dynamics of Human Learning](https://mlsanity.com/paper/2301.00901/) (2023)
- [Kaiwu: A Multimodal Manipulation Dataset and Framework for Robot Learning and Human-Robot Interaction](https://mlsanity.com/paper/2503.05231/) (2025)
- [LAMS: LLM-Driven Automatic Mode Switching for Assistive Teleoperation](https://mlsanity.com/paper/2501.08558/) (2025)
- [Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models](https://mlsanity.com/paper/2506.14727/) (2025)
- [Generalizing Multimodal Variational Methods to Sets](https://mlsanity.com/paper/2212.09918/) (2022)
- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2304.02738/) (2023)
- [Online Behavior Modification for Expressive User Control of RL-Trained Robots](https://mlsanity.com/paper/2408.16776/) (2024)
- [Learning from Humans for Proactive Assistance in Human-Robot Collaborative Transport](https://mlsanity.com/paper/2609.25351/) (2026)

## Abstract (arXiv)

We present the Human And Robot Multimodal Observations of Natural Interactive Collaboration (HARMONIC) data set. This is a large multimodal data set of human interactions with a robotic arm in a shared autonomy setting designed to imitate assistive eating. The data set provides human, robot, and environmental data views of twenty-four different people engaged in an assistive eating task with a 6 degree-of-freedom (DOF) robot arm. From each participant, we recorded video of both eyes, egocentric video from a head-mounted camera, joystick commands, electromyography from the forearm used to operate the joystick, third person stereo video, and the joint positions of the 6 DOF robot arm. Also included are several features that come as a direct result of these recordings, such as eye gaze projected onto the egocentric video, body pose, hand pose, and facial keypoints. These data streams were collected specifically because they have been shown to be closely related to human mental states and intention. This data set could be of interest to researchers studying intention prediction, human mental state modeling, and shared autonomy. Data streams are provided in a variety of formats such as video and human-readable CSV and YAML files.